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International  Science  and  Technology  for  the  Army:  Misplaced  Focus? 

COL  Michael  G.  Padgett,  USA  (Ret.) 


How  leaders  within  the  Department  of  Defense  decide  where  to  mine  for 
international  science  and  technology  (S&T)  is  more  of  a  random  process  than 
accepted  methodology.  Considering  the  importance  to  the  commercial  world 
of  optimally  located  international  research  and  development  (R&D)  centers, 
the  military  services’  current  practice  of  abdicating  the  decision-making 
process  to  a  subjective  evaluation  versus  application  of  accepted  criteria  and 
current  information  that  allows  quantification  of  the  criteria  invites  techno¬ 
logical  surprise  on  the  battlefield.  By  evaluating  each  criterion,  the  optimal 
decision  for  locating  international  S&T  mining  centers  is  possible.  Finding  the 
optimal  technologies  available  for  the  nation’s  warfighters  ensures  world-class 
technologies  for  U.S.  military  programs  and  saves  defense  funding  of  R&D  for 
developing  existent  S&T  solutions. 


Priming  the  innovation  Pump:  America  Needs  More  Scientists, 
Engineers,  and  Basic  Research 

Col  Jason  James  Denney,  USAF 


Downward  trends  in  the  number  of  U.S.  born  scientists  and  engineers,  and 
basic  research  and  development  are  threatening  U.S.  national  security  and 
economic  prosperity.  Leadership  In  science  and  technology  has  long  been  an 
unrivaled  U.S.  advantage;  however,  the  United  States  has  lost  and  Is  continuing 
to  lose  ground  In  critical  technology  metrics.  In  today’s  knowledge-based 
economy,  scientific  innovation  is  more  important  to  U.S.  economic  growth  and 
national  security  than  ever  before.  Accordingly,  the  United  States  must  rebuild 
its  foundation  of  competitiveness— its  supply  of  talented  scientists,  engineers, 
and  basic  research  and  development  resources— that  has  served  Americans  so 
well  over  the  past  50  years.  In  the  21st  century,  U.S.  success  lies  at  the  leading 
edge  of  the  scientific  frontier. 


Creating  Knowledge 


A  Statistical  Approach  to  the  Development  of  Progress  Plans  Utilizing 
Bayesian  Methods  and  Expert  Judgment 

Tiffany  L.  Lewis,  Thomas  Mazzuchi,  and  Shahram  Sarkani 

The  development  of  progress  plans  for  each  identified  technical  performance 
parameter  (TPP)  is  a  critical  element  of  technical  performance  measurement. 
The  measured  values  of  TPPs  are  referred  to  as  technical  performance  measures 
(TPMs).  These  terms  are  used  interchangeably;  how/ever,  TPMs  more  directly 
reflect  how  technical  progress  and  technical  risk  are  measured  and  evaluated. 
Progress  plans,  or  planned  performance  profiles,  are  crucial  to  effective  risk 
assessment;  however,  methods  for  developing  these  plans  are  subjective  in 
nature,  have  no  statistical  basis  or  criteria  as  a  rule,  and  are  not  sufficiently 
addressed  in  literature.  The  methodology  proposed  herein  for  progress  plan 
development  will  involve  the  elicitation  of  expert  judgments  to  formulate  prob¬ 
ability  distributions  that  reflect  the  expected  values/estimates  used  to  establish 
progress  plans. 

Analysis  of  Generation  Y  Workforce  Motivation  Using  Multiattribute 

Utility  Theory 

ian  N.  Barford  and  Patrick  T.  Hester 


This  article  explores  the  difference  in  assigned  levels  of  workplace  motivation 
and  happiness  between  federal  government  workforce  members  of  Generation 
Y  versus  Generation  X  and  Baby  Boomers.  Thirty  hypotheses  were  tested,  and 
11  were  found  to  be  statistically  significant.  Generation  Y  does  assign  different 
levels  of  importance  and  partially  assigns  different  levels  of  happiness  to  the  five 
motivational  factors  examined  in  this  study;  responsibilities,  compensation,  work 
environment,  advancement  potential,  and  free  time.  Advancement  potential  and 
free  time  were  rated  the  highest  factors  when  compared  to  Generation  X  and 
Baby  Boomers.  Sample  size  was  small  due  to  limited  availability  of  workforce 
members.  This  study  represents  the  first  attempt  to  explore  motivational  factors 
for  the  Generation  Y  workforce  within  the  federal  government. 
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Application  of  Reai  Options  Theory  to  DoD  Software  Acquisitions 

Capt  Albert  Olagbemiro,  USAFR,  Johnathan  Mun,  and  Man-Tak  Shing 


The  traditional  real  options  valuation  methodology,  when  enhanced  and  prop¬ 
erly  formulated  around  a  proposed  or  existing  software  investment  employing 
the  spiral  development  approach,  provides  a  framework  for  guiding  software 
acquisition  decision  making  by  highlighting  the  strategic  importance  of  mana¬ 
gerial  flexibility  in  managing  risk  and  balancing  a  customer’s  requirements 
within  cost  and  schedule  constraints.  This  article  discusses  and  describes  how 
an  integrated  risk  management  framework,  based  on  real  options  theory,  could 
be  used  as  an  effective  risk  management  tool  to  address  the  issue  of  require¬ 
ments  uncertainty  as  it  relates  to  software  acquisition  and  guide  the  software 
acquisition  decision-making  process. 
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From  the  Executive  Editor 


FROM  THE  EXECUTIVE  EDITOR 


This  issue  marks  the  debut  of  the  De¬ 
fense  Acquisition  Research  Journai  (ARJ).  Its 
publication  marks  a  continuation  of  the  same 
peer-reviewed  journal  that  the  Defense  Acqui¬ 
sition  University  (DAU)  has  been  publishing 
since  1994,  first  under  the  title  Defense  Ac¬ 
quisition  Review  Quarterly,  and  since  2004  as 
Defense  Acquisition  Review  Journal.  The  De¬ 
fense  ARJ  cames  forward  this  tradition  of  scholarly  excellence. 

This  change  in  name  is  part  of  an  overall  re-emphasis  on  DAU's 
research  mission.  When  the  university  was  established  by  Congress 
in  1991,  the  enabling  legislation  stated  that  it  would  “provide  for  the 
research  and  analysis  of  defense  acquisition  policy  issues  from  an 
academic  perspective”  (DAU  Structure,  1991).  DAU’s  mission  today 
supports  Department  of  Defense  (DoD)  and  congressional  initiatives 
by  providing  the  kind  of  thought  leadership  that  helps  improve  acqui¬ 
sition  outcomes.  This  journal  is  central  to  that  mission  by  providing  a 
high-quality,  peer-reviewed  forum  for  disseminating  a  broad  range 
of  research  and  analysis  from  across  the  entire  defense  acquisition 
enterprise. 

The  theme  of  this  issue,  “Creating  Knowledge,”  describes  the 
fundamental  purpose  of  acquisition  research:  to  make  sense  of  ob¬ 
servations  and  data  through  systematic  study  and  analysis,  with  the 
goal  of  creating  practical  applications  to  influence  acquisition  policies, 
procedures,  and  outcomes. 

COL  Michael  G.  Padgett,  USA  (Ret.),  leads  off  with  an  examination 
of  where  knowledge  is  created  worldwide,  and  how  the  DoD  should 
go  about  locating  and  making  best  use  of  that  knowledge.  Col  Jason 
James  Denney,  USAF,  by  contrast,  looks  at  how  the  United  States  can 
prime  the  innovation  pump  to  maintain  its  knowledge  competitiveness 
in  the  world.  Tiffany  L.  Lewis  et  al.  explain  how  expert  knowledge  and 
judgment  can  be  better  employed  in  establishing  the  progress  plans 
for  critical  program  elements.  Ian  N.  Barford  and  Patrick  T.  Hester 
look  at  how  Generation  Y— the  latest  generation  of  “knowledge  work¬ 
ers”  to  enter  the  DoD— perceives  various  motivational  factors  in  their 
workplace.  Finally,  Capt  Albert  Olagbemiro,  USAFR  et  al.  examine  the 
use  of  real  options  theory  to  improve  knowledge  and  management  of 
software  acquisition  risk. 

A  new  feature  in  the  Defense  ARJ  is  the  Defense  Acquisition  Pro¬ 
fessional  Reading  List.  The  aim  of  this  list  is  to  enrich  the  knowledge 
and  understanding  of  the  defense  acquisition  enterprise  workforce. 


The  books  that  will  be  reviewed  in  this  journal  reflect  important  histori¬ 
cal  and  contemporary  insights  that  are  directly  applicable  to  today’s 
defense  acquisition  workforce.  Leading  off  in  this  issue,  Michael  Pryce 
of  the  Manchester  Business  School  (UK)  reviews  The  Polaris  System 
Development  by  Harvey  Sapolsky.  The  methods  and  lessons  of  this 
highly  successful  project  resonate  even  after  a  half-century.  I  encour¬ 
age  Defense  ARJ  readers  to  submit  reviews  of  books,  following  the 
guidelines  set  out  in  the  Reading  List  section.  ^ 


Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 
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The  Defense  Acquisition  University  (DAU)  will  host 
its  annual  Acquisition  Connnnunity  Synnposium  on 
Tuesday,  April  12,  2011,  on  its  Fort  Belvoir  Cannpus. 
The  thenne  this  year  is: 


Through  a  series  of  speakers,  panels,  and  breakout 
sessions,  the  symposium  will  provide  congressional, 
federal-level,  Office  of  the  Secretary  of  Defense,  Ser¬ 
vice-level,  and  industry  perspectives  on  implement¬ 
ing  affordability  initiatives  announced  by  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  in  2010.  The  1-day  symposium  will  conclude 
with  a  dinner  in  the  evening  honoring  the  winner  of  the 
2011  Alumni  Association’s  Acker  Award;  induction  of 
new  DAU  Hall  of  Fame  members;  and  presentation  of 
the  annual  Hirsch  Research  Paper  Competition  prizes. 


To  register  and  for  VTC  locations, 
visit  www.dauaa.org  or  call  1-800-755-8805 
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INTERNATIONAL  SCIENCE 
AND  TECHNOLOGY 
FOR  THE  ARMY: 

MISPLACED  FOCUS? 

f  COL  Michael  G.  Padgett,  USA  (Ret.) 

How  leaders  within  the  Department  of  Defense  decide  where 
to  mine  for  international  science  and  technology  (S&T)  is  more 
of  a  random  process  than  accepted  methodology.  Considering 
the  importance  to  the  commercial  world  of  optimally  located 
international  research  and  development  (R&D)  centers,  the 
military  services’  current  practice  of  abdicating  the  decision¬ 
making  process  to  a  subjective  evaluation  versus  application 
of  accepted  criteria  and  current  information  that  allows 
quantification  of  the  criteria  invites  technological  surprise  on 
the  battlefield.  By  evaluating  each  criterion,  the  optimal  deci¬ 
sion  for  locating  international  S&T  mining  centers  is  possible. 
Finding  the  optimal  technologies  available  for  the  nation’s 
warfighters  ensures  world-class  technologies  for  U.S.  military 
programs  and  saves  defense  funding  of  R&D  for  developing 
existent  S&T  solutions. 


Keywords:  International  Technology  Center  (ITC); 
Research,  Development,  and  Engineering  Command 
(RDECOM):  Research  and  Development  (R&D); 
Science  and  Technology  (S&T);  Future  Investment 
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The  U.S.  Army  has  International  Technology  Centers  (ITCs) 
located  worldwide.  These  centers  seek  the  latest  science  and 
technology  (S&T)  across  the  globe  by  reporting  in  which  foreign 
universities,  industries,  and  government  laboratories  S&T  is  devel¬ 
oped.  Once  the  technology  is  found  and  identified  as  beneficial  to 
a  U.S.  Army  science  or  developmental  program,  certain  measures 
are  taken  to  form  a  collaborative  project  with  the  foreign  entity  to 
incorporate  the  foreign  technology  into  the  U.S.  program. 

Importance 

The  underoptimization  of  U.S.  Army  resources  is  a  possible 
outcome  for  future  decisions  if  the  Army  does  not  dedicate  scarce 
available  resources  for  mining  international  S&T  productively  (Dud¬ 
ley  &  Deylami,  2007,  pp.  44-49).  If  the  U.S.  Army  does  not  seek  the 
best  emerging  S&T,  alternative  and  competing  science  might  defeat 
existing  technology  (Daniel  &  Loeb,  2006;  National  Academy  of  Sci¬ 
ences,  2007;  Segal,  2004).  At  worst,  technological  surprise  on  the 
battlefield  might  result  in  defeat  and  destruction  of  the  American 
way  of  life  as  we  now  know  it  (Defense  Science  Board,  2007).  After 
setting  the  boundaries  of  the  least  and  worst  likely  outcomes  of 
misallocation  regarding  mining  of  international  S&T  resources,  one 
may  predict  what  is  more  likely  to  happen. 

However,  the  U.S.  Army  does  not  have  a  mature  process  for 
deciding  where  to  seek  international  S&T  that  might  render  the 
latest  worldwide  technologies  for  U.S.  warfighters  (Padgett,  2010). 
The  current  process  used  for  deciding  where  the  Army  should  seek 
beneficial,  state-of-the-art  S&T  that  will  be  incorporated  into  Army 
programs  is  not  based  on  a  systematic  analysis  of  the  countries  and 
regions  where  the  best  S&T  resides.  This  conclusion  is  based  on  a 
recent  study  that  analyzed  the  Army  Materiel  Command  (AMC) 
decision-making  process,  resulting  in  the  location  of  a  new  Army 
International  Technology  Center  (ITC)  in  Latin  America  (Padgett, 
2010),  as  well  as  a  review  of  the  literature  pertaining  to  military  and 
commercial  sector  international  S&T  research  and  development 
(R&D)  location  decisions. 

The  Navy  and  Air  Force  also  engage  in  seeking  international  S&T. 
The  Navy’s  Office  of  Naval  Research  (ONR)  has  a  central  office  in 
Great  Britain,  commanded  by  a  Navy  captain,  dedicated  to  seeking 
international  S&T  worldwide  for  the  ONR.  A  suboffice  of  the  central 
office  in  Great  Britain  is  the  Navy’s  ONR  office  located  in  the  U.S. 
Embassy  in  Santiago,  Chile.  The  ONR  office  in  Santiago  was  the  first 
Service  S&T  office  in  Latin  America.  Its  location  in  Chile  influenced 
the  Army  leadership  in  their  decision  to  co-locate  a  similar  Army 
office— ITC  Americas— in  the  U.S  Embassy-Santiago  (Padgett,  2010). 
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The  Air  Force  Office  of  Scientific  Research  (AFOSR)  is  charged 
with  seeking  internationai  S&T.  AFOSR  is  one  of  10  Air  Force 
Research  Laboratory  (AFRL)  technoiogy  directorates.  AFOSR  is 
the  oniy  AFRL  directorate  that  nnaintains  overseas  offices  for  AFRL. 
Other  U.S.  governnnent  efforts  to  seek  scientific  data  internationaiiy 
are  not  known.  Even  though  the  other  Services  may  exercise  a  more 
objective  approach  than  the  Army’s  to  their  decisions  on  S&T  min¬ 
ing  center  iocations,  the  study  found  that  there  are/were  no  wideiy 
accepted  objective  criteria  used  by  either  the  Navy  or  Air  Force 
upon  which  to  base  their  past  S&T  mining  center  location  decisions 
(Padgett,  2010;  Roth,  Perez,  Wylie,  &  Luoma,  2002). 

What  is  the  Background? 

Annual  U.S.  federal  government  expenditure  dedicated  to  R&D 
was  $129.1  billion  for  2006,  of  which  $72.1  billion  was  dedicated  to 
the  U.S.  Department  of  Defense  (DoD),  leaving  the  remaining  $57 
billion  for  the  next  five  highest  federal  departments  combined  (Got- 
tron,  2006).  The  vast  array  of  DoD  laboratories  and  program  offices 
spends  most  of  the  $72.1  billion  internally.  Resources  are  much  more 
limited  in  the  Army’s  international  search  for  S&T. 

The  U.S.  Army  spends  only  $9  million  to  fund  the  three  ITC 
regions  to  seek  technologies  worldwide  (V.  Baldwin,  personal  com¬ 
munication,  February  26,  2007).  This  is  sufficient  for  the  Army  to 
fund  the  overhead  structures  of  three  regional  centers  for  S&T  min¬ 
ing,  with  each  region  containing  two  subordinate  offices.  However, 
how  and  where  the  $9  million  is  spent  remains  critical  since  the 
budget  has  been  relatively  constant  from  2004  to  2007  and  is  not 
expected  to  increase  in  the  near  future;  and  because  the  $9  mil¬ 
lion  is  not  adequate  for  seeking  S&T  from  all  the  countries  within  a 
regional  territory. 

The  mission  of  the  Army’s  ITCs  is  to  promote  cooperation 
between  its  Research,  Development,  and  Engineering  Command 
(RDECOM)  and  international  researchers.  By  doing  so,  the  RDE- 
COM  is  made  aware  of  possible  technologies  that  might  be  of 
use  to  the  subordinate  Research,  Development,  and  Engineering 
Centers  (RDECs),  as  well  as  the  Army  Research  Laboratory.  This 
mission  is  relatively  new,  as  the  ITC  centers  were  formerly  called 
Standardization  Groups.  The  AMC  changed  the  title  and  mission 
of  the  Standardization  Groups  to  ITCs  in  2003-2004.  The  new 
mission  focused  on  investigation  and  cooperation  in  the  field  of 
applied  research,  which  is  fundamental  research  to  fully  develop 
technologies  versus  the  old  mission  that  focused  on  standardizing 
the  technologies  developed  by  the  U.S.  Army  with  our  closest  for¬ 
eign  partners  worldwide. 
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What  Research  Exists? 

To  more  specifically  review  how  the  military  has  made  past 
international  S&T  decisions,  a  study  was  made  to  examine  a  specific 
decision  made  by  the  Army  to  locate  the  newest  international  S&T 
mining  center— a  center  called  ITC  Americas.  ITC  Americas  was 
located  in  Santiago,  Chile,  with  subordinate  offices  in  Argentina 
and  Canada. 

A  review  of  literature  revealed  the  criteria  used  by  the  civilian 
sector  and  government  sources  in  making  international  S&T  loca¬ 
tion  decisions,  as  well  as  the  information  sources  used  by  the  civilian 
sector  to  compose  said  criteria  (Athukorala  &  Kohpaiboon,  2005; 
Doz,  Wilson,  Veldhoen,  &  Goldbrunner,  2006;  Economist  Intelligence 
Unit,  2004;  Goode  &  Roberts,  2004;  National  Science  Foundation, 
2006;  Organisation  for  Economic  Co-Operation  and  Development, 
2005).  The  criteria  and  information  sources  used  are  listed  in  a  later 
section  of  this  article  entitled,  “What  Does  the  Existent  Literature 
Say?”  To  augment  the  information  found  in  the  literature  review, 
the  study  resulted  in  a  survey  of  officials  within  the  RDECOM  aimed 
at  determining  the  criteria  rated  best  by  the  survey  respondents 
for  decisions  involving  the  Army’s  international  S&T  locations.  The 
survey  provided  RDECOM  officials  with  the  list  of  criteria  used  by 
the  commercial  sector  for  ranking  and  also  provided  respondents 
the  chance  to  suggest  their  own  view  of  the  most  important  criteria 
that  should  be  used  by  the  military  in  international  location  deci¬ 
sions.  As  part  of  this  section,  a  description  of  the  study  follows, 
including  the  details  of  the  study  methodology,  population,  and 
survey  instrument. 

Study  Methodology 

The  specific  problem  this  study  examined  was  to  understand, 
analyze,  and  explain  the  internal,  external,  and  political  influences 
on  the  U.S.  Army  decision-making  process  regarding  location  of  a 
U.S.  Army  S&T  center  in  Latin  America  rather  than  in  another  world 
region.  The  methodology  for  the  study  was  qualitative  and  interpre¬ 
tive.  The  design  was  a  case  study  on  the  decision  to  locate  an  S&T 
center  in  Latin  America  rather  than  other  world  regions.  One  part  of 
the  study  involved  using  some  aspects  of  the  participant  observer 
approach  by  examining  the  reports  written  by  members  of  the  ITC 
Americas.  The  survey  in  the  study  filled  in  the  gaps  in  the  literature 
reviewed  and  proposed  answers  to  the  problem  statement  and 
research  questions. 

The  purpose  of  the  qualitative  study  was  to  review  the  decision¬ 
making  process  used  by  Army  leaders  when  the  decision  was  made 
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to  locate  an  S&T  nnining  center  within  Latin  Annerica.  Fundannental 
to  the  decision  exannined  in  the  study  were  the  factors  considered 
by  senior  Arnny  leaders  in  their  valuation  of  one  world  region  over 
another.  The  study  indicated  that  the  U.S.  Arnny  R&D  leaders  did 
not  identify  and  leverage  the  nnost  appropriate  criteria  upon  which 
to  base  international  S&T  mining  decisions. 

Who  Completed  the  Study  Survey? 

The  population  selected  for  the  survey  included  30  leaders 
within  the  Army  R&D  community.  The  30  leaders  received  a  survey, 
and  the  goal  was  to  obtain  responses  from  20  of  the  30  leaders. 
Persons  involved  with  the  decision  included  both  military  and  civil¬ 
ian  members  who  were  part  of  the  Army’s  S&T  organization.  The 
survey  was  sent  to  30  DoD  leaders  believed  to  be  most  affected  by 
the  decision  to  locate  the  S&T  center  in  Latin  America.  Most  of  the 
Army  leaders  were  within  RDECOM.  Some  of  the  other  leaders  were 
within  the  Army  and  Defense  Department  secretariats. 

The  survey  participant  list  included  senior  leaders  within  RDE¬ 
COM  (n  =  5),  the  deputy  assistant  secretaries  of  the  Army  for 
Defense  Exports  and  Controls  (n  =  2),  and  the  deputy  assistant 
secretary  of  the  Army  for  Research  and  Technology  (n  =  1).  Also 
included  were  the  past  two  leaders  of  the  international  section 
for  RDECOM  (n  =  2),  leaders  within  the  international  secretariat  of 
Office  of  the  Secretary  of  Defense  (n  =  2),  and  a  past  commander 
of  the  AMC  (AMC;  n  =  1).  The  AMC  was  the  organization  with  the 
final  decision  authority  to  locate  the  S&T  office  in  Latin  America. 
The  first  group  included  13  people. 

The  second  group  of  leaders  was  within  the  subordinate  ele¬ 
ments  of  RDECOM,  called  RDECs,  and  the  Army  Medical  Command 
(MEDCOM)  (n  =  1).  The  RDEC  leaders  are  Senior  Executive  Service 
(SES)-graded  technical  directors  and  are  within  the  subordinate 
elements  of  RDECOM  (n  =  7).  The  technical  directors  in  the  subor¬ 
dinate  elements  of  RDECOM  and  MEDCOM  have  advisors,  called 
International  Points  of  Contact  (IPOCs;  n  =  9),  who  are  of  less  senior 
rank  (below  GS-15  in  grade),  and  were  also  in  the  second  group.  The 
IPOCs  are  the  principal  advisers  for  the  RDEC  technical  directors  on 
the  integration  of  international  S&T  into  Army  programs.  The  second 
group  of  leaders  included  17  people:  the  seven  technical  directors, 
the  past  deputy  commander  of  MEDCOM’s  research  center,  and  the 
nine  IPOCs  for  the  technical  directors  and  MEDCOM.  Twenty-two 
of  the  30  leaders  responded  to  the  survey. 

How  Was  the  Survey  Constructed  and  What  Did  It  Reveal? 

The  survey  instrument,  a  questionnaire  with  12  questions,  was 
created  specifically  for  the  study.  The  survey  instrument  contained 
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items  addressing  how  Army  R&D  ieaders  made  the  decision  to  seek 
quaiity  S&T  information  in  Latin  America,  the  criteria  and  informa¬ 
tion  Army  ieaders  used  to  decide  to  iocate  an  S&T  mining  activity 
within  Latin  America  versus  other  worid  regions,  and  the  criteria 
Army  ieaders  considered  when  they  decided  to  iocate  an  S&T  center 
in  Latin  America. 

The  survey  resuits  reveaied  how  the  decision  was  made  to  seek 
S&T  in  Latin  America  versus  other  worid  regions,  which  criteria  Army 
ieaders  used  to  open  the  S&T  mining  activity  within  Latin  America, 
and  the  opinions  of  the  ieaders  regarding  the  appropriateness  of 
the  criteria  proposed  in  existent  iiterature.  The  survey  also  revealed 
whether  information  currently  available  was  adequate  for  inter¬ 
national  S&T  location  decisions  and  the  opinions  of  respondents 
regarding  the  accessibility  of  S&T  in  one  emerging  world  region 
versus  another. 

What  Does  the  Existent  Literature  Say? 

In  Military  Writings/Studies 

The  Army  hired  a  contractor,  CommerceBasix,  to  study  where 
it  would  be  best  to  locate  future  ITCs  (Goode  &  Roberts,  2004). 
CommerceBasix  analyzed  the  choices  of  investing  in  Latin  America, 
India,  Japan,  China,  Singapore,  or  Eastern  Europe.  The  RDECOM 
requested  CommerceBasix  to  focus  on  at  least  two  but  no  more 
than  five  possible  options  regarding  where  to  place  international 
S&T  locations.  The  study  focused  on  considerations  for  current  and 
future  locations  for  Army  S&T  centers  worldwide.  Further,  the  study 
team  expended  much  effort  in  detailing  the  current  manning  levels 
of  the  worldwide  Army  S&T  offices,  mission  statements,  and  pub¬ 
lishing  comments  from  interviews  of  the  worldwide  S&T  offices  and 
staff  elements  within  RDECOM.  According  to  Goode  and  Roberts 
(2004),  the  study  cited  “several  direct  and  indirect  determinants 
are  commonly  believed  to  be  important  factors  in  innovation.”  The 
factors  identified  were  available  capital,  economic  life  cycle  of  a 
prospective  country,  openness  of  the  economy,  market-based  eco¬ 
nomic  systems,  protection  for  property  rights,  domestic  demand 
for  innovative  products,  and  the  balance  of  trade.  The  remainder 
of  the  study  analyzed  major  countries  of  the  world  and  how  each 
rated  considering  the  factors  selected. 

The  ONR  had  contracted  for  two  more  focused  studies.  The 
Navy  was  the  first  to  open  an  S&T  office  in  Latin  America,  locating 
their  office  in  Santiago,  Chile.  The  Navy’s  original  decision  to  locate 
their  office  in  Santiago  was  one  of  the  most  significant  factors  that 
influenced  the  Army’s  2004  decision.  The  first  Navy  study,  by  Guza 
et  al.  (2002)  used  database  searches  to  determine  what  S&T  areas 
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were  strongest  in  each  Latin  American  country.  The  reason  for  the 
study  was  to  provide  the  information  needed  to  determine  where 
within  Latin  America  an  office  shouid  be  iocated,  and  afterward, 
what  S&T  areas  shouid  be  sought  in  each  country  within  Latin  Amer¬ 
ica.  The  study  iooked  at  citations  and  S&T  articies  pubiished  within 
major  S&T  categories,  such  as  bioscience,  materiai  science,  navai 
architecture,  human  factors,  eiectronics,  computer  science,  radars, 
underwater  acoustics,  optics,  manufacturing,  and  oceanography. 
The  information  to  support  the  conclusions  of  the  study  was  derived 
from  Inspec®  Direct  and  Science  Citation  Index  (SCI)  databases. 
The  Navy  study  did  not  consider  worldwide  locations  or  selection 
criteria  for  their  recommendations  and  conclusions,  and  the  study 
was  limited  to  Latin  American  countries  (Guza  et  al.,  2002). 

The  second  Navy  study  was  conducted  by  the  Center  for  Strate¬ 
gic  Studies  (Roth  et  al.,  2002).  ONR  requested  an  assessment  of  the 
factors  that  would  indicate  the  best  location  for  a  Latin  American 
S&T  center.  The  factors  recommended  were:  economic  and  political 
considerations,  general  living  and  working  conditions,  S&T  fund¬ 
ing  levels,  and  international  cooperation  and  activities.  However, 
once  again,  this  study  only  considered  four  countries  within  Latin 
America:  Brazil,  Mexico,  Argentina,  and  Chile.  Factors  for  loca¬ 
tions  worldwide  were  not  considered  (Roth  et  al.,  2002).  The  Navy 
decided,  as  a  result  of  the  study’s  recommendations,  to  locate  their 
Latin  American  S&T  office  in  Chile. 

The  Air  Force  was  also  influenced  by  the  Navy  and  Army  deci¬ 
sions  to  locate  S&T  offices  in  Latin  America  and  opened  an  office 
in  the  U.S.  Embassy-Santiago  in  2009  (J.  Fillerup,  personal  com¬ 
munication,  February  21,  2007). 

In  the  Commercial  Sector 

Doz  et  al.  (2006)  wrote  about  the  drivers,  or  selection  criteria, 
for  innovation  and  R&D  expansion  beyond  corporate  headquarters, 
called  dispersion.  They  performed  a  survey  of  186  companies  from 
19  countries  and  17  sectors,  with  a  combined  $76  billion  in  R&D 
expenditures  in  2004.  The  number  of  foreign  R&D  sites  from  all 
countries  has  increased  from  45  percent  in  1975  to  66  percent  in 
2004.  Of  the  total  percentage  of  new  sites,  13.9  percent  of  them 
were  located  in  China  and  India.  Approximately  78  percent  of  the 
new  sites  remain  in  the  United  States  or  Europe.  The  remaining 
8  percent  will  go  into  other  Asian  and  Latin  American  countries. 
The  forecast  was  that  by  the  end  of  2007,  China  would  have  31 
percent  of  the  global  R&D  staff— up  from  19  percent  in  2004.  The 
study  predicted  that  the  number  of  R&D  sites  and  staff  in  Western 
Europe  and  the  United  States  would  remain  unchanged.  The  Doz 
report  cited  the  percentage  of  new  sites:  China-22  percent;  United 
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FIGURE  1.  ECONOMIST  INTELLIGENCE  UNIT  SURVEY 
QUESTION-FUTURE  INVESTMENT  TOP  10  LOCATIONS 


In  which  of  the  following 
countries  does  your  company 


plan  to  spend  the  most  on  R&D 
in  the  next  3  years  (excluding 
your  domestic  market)  (top  10 
locations  out  of  54)? 

1. 

China 

39 

2. 

United  States 

29 

3. 

India 

28 

4. 

United  Kingdom 

24 

5. 

Germany 

19 

6. 

Brazil 

11 

7. 

Japan 

10 

8. 

France/Italy 

9 

10. 

Czech  Republic 

8 

FIGURE  2.  ECONOMIST  INTELLIGENCE  UNIT  SURVEY 
QUESTION-LOCATION  OF  CORPORATE  HEADQUARTERS 

Latin  America 
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FIGURE  3.  ECONOMIST  INTELLIGENCE  UNIT  SURVEY 
QUESTION-OVERSEAS  R&D  EXPENDITURE  OVER  PAST  3  YEARS 


Roughly  what  percentage  of  your  company's 
overseas  R&D  expenditure  over  the  past  3 
years  went  to  the  following  regions? 

Under  10% 

Over  10% 

Latin  America 

86% 

14% 

Eastern  Europe 

78% 

24% 

Asia 

50% 

50% 

Note.  Due  to  rounding  issues  in  the  source  from  which  the  data  was  adapted,  not  all 
percentages  add  up  to  100%. 


FIGURE  4.  ECONOMIST  INTELLIGENCE  UNIT  SURVEY 
QUESTION-ANTICIPATED  OVERSEAS  R&D  EXPENDITURE  OVER 
NEXT  3  YEARS 


Roughly  what  percentage  of  your  company's 
overseas  R&D  investment  will  be  allocated  to 
the  following  regions  in  the  next  3  years? 

Under  10% 

Over  10% 

Latin  America  83% 

16% 

Eastern  Europe  65% 

36% 

Asia  38% 

62% 

Note.  Adapted  from  Scattering  the  Seeds  of  Invention:  The  Globalisation  of  Research  and 
Development,  by  Economist  Intelligence  Unit,  2004.  Due  to  rounding  issues  in  the  source 
from  which  the  data  was  adapted,  not  all  percentages  add  up  to  100%. 


States  and  India-19  percent;  Western  Europe-13  percent;  Eastern 
Europe-12  percent;  Asia,  excluding  China-8  percent;  and  Latin 
America-S  percent.  The  report  said  the  shift  of  R&D  sites  is  towards 
India  and  China. 

The  Organisation  for  Econonnic  Co-Operation  and  Developnnent 
(OECD)  (2005)  also  proposed  selection  criteria  for  R&D  locations: 
patents,  licenses,  know  how,  R&D  studies,  trade  in  high-tech  prod¬ 
ucts,  and  protection  of  intellectual  property.  Rausch  (2003),  the 
World  Intellectual  Property  Organization  (2006),  and  two  other 
OECD  reports  (2008a;  2008b)  added  to  the  innportance  of  patents 
as  an  indicator  of  S&T.  The  OECD  2008  reports  said  patent  statis¬ 
tics  measure  the  output  of  R&D  and  its  productivity.  The  reports 
said  patenting  activity  is  more  concentrated  than  R&D  effort,  and 
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therefore  is  a  better  measure  of  a  country’s  R&D  activity.  The  OECD 
report  (2008b)  aiso  said  triadic  patents  are  the  best  measure  since 
they  better  refiect  the  quaiity  of  patents.  Triadic  are  those  patents 
fiied  in  the  United  States  Patent  Office,  Japan  Patent  Office,  and 
European  Patent  Office. 

A  study  by  the  Economist  Inteiiigence  Unit  (2004)  found  that 
cheap  iabor  was  an  insignificant  factor  in  R&D  iocation  decisions. 
The  study  done  by  the  Economist  Inteiiigence  Unit  included  104 
senior  executives.  It  found  that  52  percent  of  corporate  executives 
plan  to  increase  their  investment  in  overseas  research  in  the  next  3 
years,  primarily  in  China  and  India.  The  study  also  addressed  devel¬ 
opments  \A/ithin  Eastern  Europe.  The  study  said  the  relative  skill  sets 
of  Eastern  Europe  are  increasing,  \A/hile  Western  Europe  and  the 
United  States  are  declining,  making  Eastern  Europe  an  attractive 
location  for  future  R&D  investment.  Four  questions  were  posed  to 
executives;  Figures  1  through  4  reflect  their  ans\A/ers. 

A  study  by  Thursby  and  Thursby  (2006)  surveyed  over  200 
multinational  companies  across  15  industries  regarding  factors 
influencing  location  decisions  for  R&D  facilities.  The  majority  of 
companies  surveyed  were  in  the  United  States  and  Western  Europe. 
The  study  was  done  out  of  concern  that  location  decisions  should 
be  based  on  informed  versus  anecdotal  data.  The  search  used  61 
articles,  dating  from  2002-2005,  from  the  New  York  Times  and  Wall 
Street  Jouma/ describing  R&D  location  decisions.  Of  the  articles,  38 
mentioned  cost  as  the  primary  deciding  factor  and  29  the  quality  of 
R&D  personnel.  Figure  5  sho\A/s  the  increases  and  decreases  in  R&D 
employment  in  several  \A/orld  regions  and  countries. 

India  and  China  reflected  the  largest  gains  by  far.  The  Thursby 
study  also  assessed  the  locations  of  U.S.  and  Western  European 
sites  recently  opened  or  future  sites  the  two  regions  may  intend  to 
open  (Figure  6). 

Cimoli,  Ferraz,  and  Primi  (2005)  found  Latin  America  to  be  a 
minor  actor  in  patenting  activity,  and  the  patents  filed  are  usually 
in  chemicals  and  mechanics,  not  technology-leading  areas  such 
as  telecommunications,  biotechnology,  genetics,  and  electronics. 
The  report  said  Latin  America  innovative  processes  are  adaptive  in 
nature  and  are  rarely  inventions  and  scientific  discoveries.  Cimoli  et 
al.  also  cited  Latin  American  technology  policy  as  follo\A/ing  diver¬ 
gent  and  unsynchronized  patterns. 

The  data  in  an  OECD  Report  (2008b)  reflect  patent  information 
designed  to  indicate  innovative  activity.  As  depicted  in  Figure  7,  the 
United  States  has  the  top  innovative  performance  at  36.4  percent 
of  the  total,  follo\A/ed  by  the  European  Union  (30.3  percent)  and 
Japan  (25.7  percent).  Changes  that  were  cited  as  a  measure  of  pat¬ 
ent  quality  in  the  triadic  patent  top  20  indicate  innovation  in  Asia  is 
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FIGURE  5.  REGIONS/COUNTRIES  WHERE  AN  INCREASE/ 
DECREASE  IN  R&D  EMPLOYMENT  IS  ANTICIPATED 
General  Background 


80 


United 

Western 

Former 

China 

India 

Other 

States 

Europe 

Soviet  Bloc 

Note.  Adapted  from  Here  or  There?  A  Survey  of  Factors  in  Multinational  R&D  Locations,  by  J. 
Thursby  and  M.  Thursby,  2006. 


FIGURE  6.  LOCATION  OF  RECENT  OR  PLANNED  INTERNATIONAL 
TECHNOLOGY  CENTERS 


Destination 

Home 

United 

Western 

Row 

Country 

States 

Europe 

China 

India 

Other 

Total 

United  States 

0 

19 

30 

9 

13 

71 

Western 

Europe 

14 

10 

23 

9 

12 

68 

Other 

0 

0 

2 

0 

2 

4 

Column  Total 

14 

29 

55 

18 

27 

143 

Note.  Adapted  from  Here  or  There?  A  Survey  of  Factors  in  Multinational  R&D  Locations,  by  J. 
Thursby  and  M.  Thursby,  2006. 


surging.  No  Latin  American  country  is  inciuded  in  the  top  30,  in  raw 
numbers  or  as  compared  to  popuiation  or  gross  domestic  product 
(GDP).  Three  countries  from  Centrai/Eastern  Europe  are  in  the  top 
30  of  triadic  patents  as  compared  with  GDP  and  popuiation  density: 
the  Russian  Federation,  Czech  Republic,  and  Hungary. 

Pion-Berlin  (2005)  said  that  Latin  America,  unlike  Eastern 
Europe,  in  the  past  has  not  encountered  a  security  threat  sufficient 
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FIGURE  7.  SHARE  OF  COUNTRIES  IN  TOTAL  TRIADIC  PATENT 
FAMILIES,  2003 


1.4% 


Note.  Patent  counts  are  based  on  the  earliest  priority  date,  the  inventor's  country  of  residence, 
and  fractional  counts.  Adapted  from  WlPO  Patent  Report:  Statistics  on  Worldwide  Patent 
Activities,  by  World  Intellectual  Property  Organization,  2006;  and  Compendium  of  Patent 
Statistics,  by  Organisation  for  Economic  Co-Operation  and  Development,  2008a. 


to  motivate  an  investment  in  resources  and  taient  to  create  sophis¬ 
ticated  war  machines  nor  “civiiian  overseers”  that  understand  how 
sophisticated  war  machines  can  work  for  politicai  purposes.  The 
expenditure  toward  defense  within  Latin  America  is  iess  than  West¬ 
ern  Europe,  Asia,  Africa,  and  North  America. 

D.  Hiii  pubiished  a  report  (2002)  that  provided  an  overaii  assess¬ 
ment  of  various  regions/countries  (Figure  8).  The  technologicai 
infrastructure  is  viewed  as  the  singie  most  important  item  in  decid¬ 
ing  iocations  for  internationai  S&T  iocations. 

A  study  by  Athukoraia  and  Kohpaiboon  (2005)  said  that 
deveioped  countries  constitute  around  90  percent  of  the  U.S. 
Muitinationai  Enterprise  (MNE)  R&D  investment,  down  from  94 
percent  in  the  eariy  1990s.  According  to  the  study,  the  increase 
is  going  to  Asia,  especiaiiy  Singapore,  Korea,  Maiaysia,  and  China. 
Latin  America,  except  Mexico,  has  declined  in  recent  years  in  U.S. 
MNE  R&D  investment.  These  statistics  add  to  the  evidence  that 
locating  an  Army  S&T  center  in  Latin  America  was  not  based  on 
existent  statistics  of  locations  experiencing  the  greatest  level  of 
S&T  activity  internationally. 

Freeman  (2005)  published  a  study  citing  locations  where  sci¬ 
ence  and  engineering  (S&E)  expertise  in  the  form  of  university 
graduates  lies  worldwide.  The  study  cited  PhDs  in  S&E  as  the  most 
critical  indicator  of  where  knowledge  expertise  in  S&E  lies.  The  top 
five  countries  where  firms  intend  to  increase  R&D  efforts  outside  of 
their  homeland  were  the  United  States,  India,  the  United  Kingdom, 
and  Germany,  with  the  greatest  growth  rate  of  S&Es  predicted  to  be 
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FIGURE  8.  INDICATORS  OF  TECHNOLOGICAL  COMPETITIVENESS: 
1999  (INDEX) 


Higher  Numbers  =  Increased  Competitiveness 

Region/ 

National 

Socioeconomic  Technological 

Productive 

Country 

Orientation^ 

Infrastructure'’ 

Infrastructure' 

Capacity’' 

Latin  America 

Argentina 

41.3 

53.3 

27.5 

31.0 

Brazil 

61.5 

49.1 

40.4 

39.6 

Mexico 

41.8 

40.4 

21.8 

24.8 

Venezuela 

39.8 

49.4 

21.3 

24.3 

East  Asia 

China 

65.3 

52.4 

46.4 

41.9 

Indonesia 

53.9 

43.8 

19.2 

23.7 

Malaysia 

69.5 

58.9 

31.9 

44.1 

Philippines 

60.9 

63.7 

24.4 

42.6 

South  Korea 

74.9 

73.5 

44.6 

48.8 

Thailand 

50.7 

46.5 

20.5 

30.6 

India 

67.7 

48.4 

46.8 

51.3 

Poland 

69.6 

58.4 

38.2 

44.3 

South  Africa 

50.2 

53.6 

40.5 

28.7 

Note.  Adapted  from  Latin  America:  High-Tech  Manufacturing  on  the  Rise,  but  Outpaced  by  East 
Asia  (NSF  Publication  No.  02-331),  by  D.  Hill,  2002,  National  Science  Foundation,  Directorate 
for  Social,  Behavioral,  and  Economic  Sciences. 

®  National  orientation  provides  evidence  that  a  nation  is  taking  direct  action  to  achieve 
technological  competitiveness. 

Socioeconomic  infrastructure  assesses  the  social  and  economic  institutions  that  support  a 
modern  technology-based  industrial  nation. 

Technological  infrastructure  assesses  the  institutions  and  resources  that  contribute  to  high 
technological  development. 

^  Productive  capacity  assesses  the  level  and  efficiency  of  physical  and  human  resources 
devoted  to  manufacturing. 


in  China  by  2010.  Murdock  (2005)  nneasured  ieadership  in  R&D  as 
strength  in  investment,  scientific  pubiications,  and  patents.  Figures 
9  and  10  iiiustrate  the  percentage  of  giobai  R&D  investment. 

To  put  this  in  perspective,  Segal  (2004)  said  that  no  one  mea¬ 
sure  can  measure  a  country’s  innovation.  This  is  offered  to  sho\A/  that 
location  criteria  for  R&D  location  decisions  found  in  commercial  and 
government  studies  vary  from  study  to  study. 
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FIGURE  9.  U.S.  GOVERNMENT  SHARE  OF  GLOBAL  R&D 
INVESTMENT 

Share  of  Global  Investment 


The  United  States 
currently  leads  the  world 
in  government  R&D 
investment,  with  a  little 
over  25  percent  of  the 
total  global  investment. 


Note.  Adapted  from  Nanotechnology:  Where  Does  the  U.S.  Stand?  by  S.  Murdock,  2005. 
Testimony  of  Sean  Murdock,  Executive  Director,  Nanobusiness  Alliance,  Hearing  before 
the  Research  Subcommittee  on  the  House  Committee  on  Science. 

What  Answers  Emerged  from  the  Survey? 

Survey  respondents  thought  that  the  Navy  presence  in  Latin 
America,  prior  to  the  decision  to  iocate  an  Army  office,  affected 
the  outcome  of  the  decision.  The  consoiidation  of  scarce  miiitary 
S&T  assets  might  appear  to  be  optimai,  but  \A/hat  if  the  quantitative 
data  indicate  a  different  iocation  \A/ould  be  more  productive  than 
the  eariier  location  decision  made  by  a  different  Service? 

Some  survey  respondents  thought  that  the  decision  might 
have  been  made  to  avoid  technological  surprise  on  the  battle¬ 
field.  The  Army  decision  authority,  \A/ho  responded  to  the  survey, 
did  not  mention  avoiding  technological  surprise  as  a  factor  that 
influenced  his  decision.  Hovyever,  the  decision  authority  did  say 
that  the  prior  absence  of  any  Army  S&T  coverage  of  Latin  America 
affected  his  decision. 

Are  There  Answers  Resulting  from  the  Literature 
Review  and  Survey? 

Conclusions 

No.  1.  The  2008  OECD  Report  indicated  that  triadic  patents  \A/ere 
the  single  most  important  criterion  for  selecting  international  S&T 
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FIGURE  10.  R&D  INTENSITY  IN  DEVELOPING  COUNTRIES  AND 
SELECTED  COUNTRIES/REGIONS  IN  ASIA:  1990-2000 
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mining  iocations  (OECD,  2008b).  The  reason  triadic  patents  were 
viewed  as  an  important  criterion  was  based  on  the  view  that  if  a 
patent  was  of  great  vaiue,  the  originator  wouid  want  to  protect  it 
woridwide  as  much  as  possibie.  Aithough  this  decision  contradicts 
the  iiterature  of  patents— peer-reviewed  articles,  intellectual 
property  rights  protection,  among  a  list  of  other  things— it  is  a 
logical  one.  Knowledge  is  important,  but  the  statistics  reflect  that 
the  greatest  numbers  of  S&Es  are  in  China  and  India,  and  R&D 
intensity  is  not  as  important  as  raw  R&D  expenditures.  The  latter 
provides  the  degree  and  volume  of  opportunities  for  the  military. 

No.  2.  Based  on  the  data  reviewed  from  commercial  and  military 
sources  found  in  the  literature,  the  decision  made  to  locate  an 
Army  S&T  center  in  Latin  America  versus  another  world  region  was 
not  the  most  optimal  decision  for  the  best  long-term  interests  of 
the  U.S.  Army.  Instead,  the  data  indicated  that  the  office  should 
have  been  located  in  the  following  locations,  in  order  of  priority: 

1.  India 

2.  Emerging  Asian  countries 

3.  Eastern  Europe 

China  would  be  the  No.  1  priority,  but  its  choice  as  a  viable  loca¬ 
tion  for  an  ITC  is  viewed  as  impractical  due  to  the  need  to  protect 
U.S.  Army  intellectual  property. 
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No.  3.  The  decision  authority  for  the  S&T  center  in  Latin  America 
made  the  decision  based  on  a  personai  evaiuation;  therefore,  the 
decision  was  an  intuitive-based  one,  reiying  on  existing  information 
without  seeking  input  from  his  subordinate  expert  advisors 
(Padgett,  2010). 

No.  4.  The  survey  responses  indicated  the  decision  authority  did 
not  seek  the  expert  knowledge  of  center  directors  at  the  subordinate 
research  centers  within  RDECOM,  even  though  they  were  general 
officer-level  civilians,  prior  to  making  the  decision  (Padgett,  2010). 

No.  5.  The  decision  authority  did  not  use  a  set  of  criteria  for  the 
decision  or  seek  information  that  might  support  some  set  of  criteria, 
but  instead  evaluated  the  existent  information  subjectively  rather 
than  objectively  prior  to  reaching  the  decision  (Padgett,  2010). 

No.  6.  Overriding  factors,  three  of  which  evolved  as  a  result  of  the 
study,  motivated  the  decision  authority  to  select  Latin  America  as  the 
location  of  the  S&T  mining  center.  Navy  presence  in  Latin  America 
prior  to  the  decision  to  locate  an  Army  center  affected  the  outcome 
of  the  decision.  This  was  followed  by  quantitative  data  that  indicate 
a  different  location  would  be  more  suitable  in  the  long  term.  The 
data  indicate  that  locating  an  S&T  center  in  a  location  where  other 
S&T  assets  are  already  located  to  gain  short-term  efficiencies  from 
the  collocation  is  not  the  best  alternative  for  long-term  corporate, 
military,  or  organizational  health. 

No.  7.  The  Army  decision  authority  used  other  factors  to 
determine  where  to  locate  the  S&T  center,  such  as  politics  and 
security  cooperation  goals  for  the  military.  Survey  respondents 
cited  each  of  these  factors  repetitively.  According  to  the  survey 
responses,  political  considerations  constituted  the  largest  influence 
over  the  decision  authority  in  his  evaluation  of  the  S&T  center’s 
eventual  location. 

Recommendations 

No.  1.  Decisions  for  international  military  S&T  locations  should  use 
the  following  three  objective  criteria  as  the  basis  for  the  decision: 

1.  Triadic  patents 

2.  R&D  expenditure  rates 

3.  S&E  articles  published 

The  second  criterion  was  cited  as  the  most  important  factor  in 
a  study  by  Athukorala  and  Kohpaiboon  (2002).  The  third  criterion 
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appears  as  the  statistical  factor  used  to  measure  international  S&T 
activity  by  the  National  Science  Foundation  statisticians  in  their 
periodic  reports  (Hill,  2002,  2004). 

No.  2.  S  ince  a  subordinate  office  of  the  regional  center  in  Japan, 
located  in  Singapore,  is  presently  charged  \A/ith  oversight  of  India, 
perhaps  a  transfer  of  assets  from  the  ITC  in  London  to  Singapore 
to  better  cover  the  explosion  of  science  in  India  \A/ould  present  the 
best  option.  ITC  London  has  the  most  international  S&T  resources  at 
present,  and  those  resources  are  covering  an  area  \A/here  the  Army 
S&T  community  already  has  strong  ties,  meaning  the  resources  in 
ITC  London  might  be  better  utilized  to  detect  ne\A/  technologies 
emerging  from  India  and  the  emerging  Asian  countries.  The  same 
realignment,  using  ITC  London  assets,  should  be  applied  for  the 
emerging  Asian  countries,  except  that  the  additional  assets  should 
be  located  in  Japan  versus  Singapore  for  coverage  of  the  emerging 
Asian  countries. 

No.  3.  Based  on  using  objective  criteria,  the  office  in  Latin  America 
should  be  transferred  to  Eastern  Europe  to  cover  the  larger  amount 
of  S&T  emerging  from  Eastern  Europe.  Ho\A/ever,  if  political  concerns 
are  overriding  in  motivating  relationship  building  \A/ith  Latin  America 
versus  Eastern  Europe,  then  the  S&T  regional  office  and  suboffices 
in  the  Americas  should  remain  in  place. 

No.  4.  Senior-level  decision  authorities  should  seek  the  expert 
judgment  of  the  SES  directors  prior  to  making  international 
location  decisions. 

No.  5.  An  earlier  study  indicates  the  decision  authority  opted  to 
exclude  from  consideration  the  objective  data  and  input  from  experts 
\A/hen  he  decided  to  locate  an  S&T  center  in  Latin  America  (Padgett, 
2010).  If  a  decision  authority  opts  to  exclude  from  consideration  the 
existent  objective  data  or  the  advice  of  subordinate  experts,  then 
the  decision  authority  should  clearly  articulate  \A/hy  the  alternative 
solution  provides  a  more  optimal  outcome. 
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PRIMING  THE 
INNOVATION  PUMP: 

AMERICA  NEEDS  MORE 
SCIENTISTS,  ENGINEERS, 
AND  BASIC  RESEARCH 


1  Co/  Jason  James  Denney,  USAF 

Downward  trends  in  the  number  of  U.S.  born  scientists  and 
engineers,  and  basic  research  and  development  are  threatening 
U.S.  national  security  and  economic  prosperity.  Leadership  in 
science  and  technology  has  long  been  an  unrivaled  U.S.  advan¬ 
tage;  however,  the  United  States  has  lost  and  is  continuing  to 
lose  ground  in  critical  technology  metrics.  In  today’s  knowledge- 
based  economy,  scientific  innovation  is  more  important  to 
U.S.  economic  growth  and  national  security  than  ever  before. 
Accordingly,  the  United  States  must  rebuild  its  foundation  of 
competitiveness— its  supply  of  talented  scientists,  engineers, 
and  basic  research  and  development  resources— that  has  served 
Americans  so  well  over  the  past  50  years.  In  the  21st  century, 
U.S.  success  lies  at  the  leading  edge  of  the  scientific  frontier. 


Keywords:  Scientists,  Engineers,  Research,  Economic 
Growth,  National  Security 
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The  United  States  has  led  the  world  in  science,  technology, 
knowledge  generation,  and  innovation;  however,  the  nation  can 
no  longer  take  its  suprennacy  for  granted.  Nations  fueled  by  glo¬ 
balization  and  connpetitiveness  are  on  a  fast  track  to  surpass  the 
United  States  in  scientific  excellence  and  technological  innovation. 
Specifically,  downward  trends  in  the  nunnber  of  U.S.  scientists  and 
engineers  (S&Es),  and  national  basic  research  and  developnnent 
(R&D)  are  exacerbating  this  challenge  and  have  troubling  innplica- 
tions  for  U.S.  economic  prosperity  and  national  security.  Increasing 
the  U.S.  supply  of  quality  S&Es  and  boosting  basic  R&D  resources 
are  essential  to  national  security  and  economic  growth.  Continued 
economic  and  national  security  requires  effective  industry  and 
government  action  as  well  as  policies  to  ensure  the  United  States 
remains  at  the  leading  edge  of  the  scientific  frontier. 

From  Producing  Stuff  to  Producing  Ideas 

The  liberal,  neo-classical  economic  doctrine  and  its  principle 
that  capital  drives  growth  has  given  way  to  knowledge  economics 
and  its  principle  that  innovation  drives  growth  (Atkinson,  2009, 
slides  40-41).  During  the  industrial  revolution,  physical  capital  was 
the  competitive  advantage  and  growth  was  the  product  of  land, 
labor,  and  capital— in  other  words,  how  much  “stuff”  was  produced. 
In  1930,  Joseph  Schumpeter,  an  Austrian  economist,  first  pointed 
out  that  innovation  is  the  key  to  economic  growth  (Schumpeter, 
1930).  Today,  Paul  Romer’s  new  growth  theory  builds  on  Schum¬ 
peter’s  premise  by  stressing  that  information  leads  to  knowledge 
and  then  knowledge  leads  to  innovation  {The  Knowledge  Economy, 
2009,  pp.  4-5).  Value  is  created  by  combining  information  and 
knowledge  into  new  combinations— what  Romer  calls  recipes.  A 
recipe  is  more  valuable  than  its  parts;  and  new  combinations  are 
limited  only  by  a  person’s,  a  corporation’s,  or  a  nation’s  ability  to 
innovate  (Romer,  2007). 

Knowledge  quickly  becomes  obsolete  in  a  globalized  environ¬ 
ment,  so  competitive  advantage— based  on  knowledge— requires 
the  continuous  creation  of  more  knowledge  into  innovative  prod¬ 
ucts  and  processes.  This  framework  applies  to  both  economic  and 
national  security.  The  pendulum  has  swung  such  that  poor  countries 
of  the  future  will  have  no  ideas,  whereas  poor  countries  of  the  past 
had  no  natural  resources.  Taiwan,  for  example,  started  with  essen¬ 
tially  no  natural  resources  but  grew  rapidly  because  of  its  ability  to 
innovate.  The  U.S.  innovation  engine— fueled  by  the  supply  of  S&Es 
and  basic  R&D  resources— is  quickly  losing  ground  to  international 
competitors  that  are  rapidly  accumulating  intellectual  capital  and 
R&D  capacity.  Of  specific  concern  is  the  general  lack  of  interest 
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FIGURE  1.  U.S.  TRADE  BALANCE  IN  HIGH-TECHNOLOGY  GOODS: 
2000-2006 
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Note.  Adapted  from  Science  and  Engineering  Indicators  2008,  by  National  Science 
Board,  2008,  NSB  Report  No.  08-01. 


among  American-born  youth  in  pursuing  careers  in  the  science, 
technology,  engineering,  and  math  (STEM)  fields;  and  the  long-term 
decline  in  the  national  investment  in  basic  R&D  (Marshall,  Coffey, 
Saalfeld,  &  Col\A/ell,  2004,  p.  1).  If  these  trends  continue,  the  United 
States  \A/ill  find  itself  at  a  severe  disadvantage. 

Losing  Ground  on  All  Fronts 

By  most  science  and  technology  (S&T)  metrics  the  United  States 
leads  the  \A/orld.  Ho\A/ever,  the  nation  has  already  lost  and  is  continu¬ 
ing  to  lose  ground  in  critical  technology  output  metrics  such  as  its 
trade  balance  of  high-tech  goods  (Figure  1),  the  number  of  techni¬ 
cal  articles  published,  and  the  number  of  technical  articles  cited  by 
others  (National  Science  Board  [NSB],  2008,  pp.  10-12).  In  a  recent 
study,  38  of  the  \A/orld’s  50  leading  research  institutions  were  in 
the  United  States;  ho\A/ever,  other  nations  are  quickly  catching  up 
(Freeman,  2006,  pp.  2-3).  For  example,  multinational  companies 
are  operating  53  state-of-the-art,  high-tech  industrial  parks  and  750 
R&D  centers  in  China  (Berry  &  Loeb,  2008,  p.  6).  Gro\A/th  in  overseas 
R&D  infrastructure  has  increased  the  off-shoring  of  U.S.  industrial 
R&D  (Figure  2)  (Atkinson,  2009,  slide  25).  In  addition,  the  \A/orld’s 
S&T  investment  increased  by  96  percent  from  1996  to  2006,  \A/ith 
China’s  gro\A/th  at  9  percent,  d\A/arfing  all  other  countries,  including 
the  United  States  \A/hose  S&T  investment  decreased  by  6  percent 
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FIGURE  2.  BOOM  IN  RESEARCH  AND  DEVELOPMENT 
PERFORMED  OVERSEAS  BY  U.S.  COMPANIES 
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Note.  Adapted  from  Innovation  and  U.S.  Economic  Growth,  by  R.  D.  Atkinson,  February 
5,  2009,  Information  Technology  and  Innovation  Foundation  presentation  to  Industrial 
College  of  the  Armed  Forces. 


(Berry  &  Loeb,  2008,  p.  2).  As  a  result,  China  “isn’t  just  making 
T-shirts  anymore,”  it  is  producing  increasing  amounts  of  medium-  to 
high-tech  products  for  both  commercial  and  military  use  (Figure  3), 
and  other  countries  are  following  suit  (Atkinson,  2009,  slide  29). 
The  U.S.  output  of  native-born  S&Es,  however,  is  just  as  worrisome. 

The  United  States  lags  behind  global  competitors  in  the  percent¬ 
age  of  undergraduates  earning  S&E  degrees  (Figure  4)  (Atkinson, 
2009,  slide  26).  In  2002,  only  17  percent  of  U.S.  undergraduates 
earned  engineering  degrees,  as  compared  to  53  percent  in  China 
(National  Science  Foundation  [NSF],  2006,  Appendix,  Table  2-38). 
In  addition,  the  U.S.  global  share  of  S&E  doctorates  and  undergradu¬ 
ate  degrees  fell  from  40  to  20  percent  and  from  30  to  14  percent 
between  1970  and  2000  (Freeman,  2006,  pp.  2-3).  According  to 
the  NSF,  58  percent  of  engineering  doctorates  awarded  in  the 
United  States  in  2003  went  to  noncitizens,  while  greater  than  half 
of  the  students  enrolled  in  U.S. -taught  engineering  programs  were 
foreign-born.  And  in  2004,  S&E  doctorates  awarded  to  temporary 
residents  increased  by  9  percent,  compared  to  2  percent  for  U.S. 
citizens  (National  Defense  Education  Program  [NDEP],  2009a,  p.  2). 
Also,  in  a  recent  survey  of  more  than  270,000  U.S.  college  freshmen, 
only  7.5  percent  said  they  intended  to  major  in  engineering— the 
lowest  level  since  the  1970s  (Aerospace  Industries  Association  [AIA], 
2008,  p.  4).  While  foreign  innovations  benefit  the  standard  of  living 
in  the  United  States,  the  government  must  increase  its  own  supply 
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FIGURE  3.  CHINA  ISN'T  JUST  MAKING  T-SHIRTS  ANYMORE 
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Note.  Adapted  from  Innovation  and  U.S.  Economic  Growth,  by  R.  D.  Atkinson,  February 
5,  2009,  Information  Technology  and  Innovation  Foundation  presentation  to  Industrial 
College  of  the  Armed  Forces. 


FIGURE  4.  UNITED  STATES  LAGS  IN  PERCENT  OF  UNIVERSITY 
STUDENTS  RECEIVING  DEGREES  IN  SCIENCE  AND  ENGINEERING 
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Note.  Adapted  from  Innovation  and  U.S.  Economic  Growth,  by  R.  D.  Atkinson,  February 
5,  2009,  Information  Technology  and  Innovation  Foundation  presentation  to  Industrial 
College  of  the  Armed  Forces. 


30 


A  Publication  of  the  Defense  Acquisition  University 


http://www.clau.mil 


of  S&Es  and  basic  R&D  resources  to  nnaintain  its  edge  in  econonnic 
and  nationai  security  matters. 

Since  over  haif  the  American-born  S&E  \A/orkforce  is  over  40 
and  \A/iii  retire  in  the  next  20-30  years,  the  increasing  number  of 
foreign-born  versus  U.S.-born  S&E  students  exacerbates  the  eco¬ 
nomic  and  nationai  security  diiemma  (Marshaii  et  ai.,  2004,  p.  3). 
The  Department  of  Defense  (DoD)  aione  is  expected  to  lose  more 
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HOW  CAN  THE  UNITED  STATES  STOP  THIS  DOWNWARD 
SPIRAL  IN  S&E  AND  BASIC  R&D  CAPACITY? 
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than  13,000  S&Es  in  the  next  decade  (NDEP,  2009b,  p.  2).  Industry 
is  not  immune  either.  Sixty  percent  of  the  aerospace  industry  \A/ork- 
force  is  45  or  older,  and  27  percent  of  its  engineering  \A/orkforce  is 
qualified  for  retirement  (AIA,  2008,  pp.  3-4).  Foreign-born  S&Es 
are  earning  the  lion’s  share  of  undergraduate  and  graduate  S&E 
degrees,  but  security  concerns  \A/ith  foreign-born  S&Es  limit  their 
opportunities  \A/ithin  the  DoD  and  its  supporting  contractors  as  \A/ell 
as  other  federal  agencies  (Marshall  et  al.,  2004).  Ho\a/  can  the  United 
States  stop  this  do\A/n\A/ard  spiral  in  S&E  and  basic  R&D  capacity? 

The  U.S.  Innovation  Engine- 
Running  Lean  on  S&Es  and  Basic  R&D 

U.S.  investment  in  the  physical  sciences,  engineering,  and  R&D 
has  not  kept  pace  \A/ith  demands  of  the  global  economy  and  national 
security  threats.  September  11,  2001,  and  its  continuing  aftermath 
underlie  the  need  for  a  po\A/erful  U.S.  S&T  effort;  ho\A/ever,  the  num¬ 
ber  of  U.S.  citizens  enrolling  in  graduate  math,  engineering,  and 
physical  science  programs— the  fields  of  broadest  DoD  application- 
fell  by  25,  21,  and  17  percent  (National  Science  Foundation  [NSF], 
2001).  In  addition,  70  percent  of  the  \A/orld’s  R&D  is  now  conducted 
outside  the  United  States  (Rees,  2008a,  slide  5).  Ho\a/  can  this  be 
considering  96  percent  of  Americans  believe  S&T  plays  a  significant 
role  in  national  security,  80  percent  believe  S&T  is  very  important 
to  meeting  future  terrorist  threats,  and  90  percent  are  concerned 
that  lo\A/  S&T  performance  \A/ill  impact  the  nation’s  future  economic 
prosperity  (Bayer  Corporation,  2003)?  Actions  need  to  match  senti¬ 
ment  for  a  shift  to  occur. 

S&T  innovation  fuels  the  product  development  cycle.  Ho\a/- 
ever,  the  DoD,  particularly  over  the  last  20  years,  is  expending 
extraordinary  energy  attempting  to  incrementally  improve  existing 
capabilities,  resulting  in  diminishing  gains  in  capability  at  excessive 
cost  (Chao,  2009,  slide  7).  This  shortsighted  approach  places  the 
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U.S.  military  dominance  and  economic  competitiveness,  particularly 
in  the  defense  sector,  at  risk.  “Current  military  dominance  derives 
from  S&T  investments  made  in  the  1950s  through  the  1970s  by  DoD 
and  other  federal  agencies”;  therefore,  shortsightedness  today  may 
concede  U.S.  military  dominance  10,  20,  and  up  to  30  years  or  more 
in  the  future  (Marshall  et  al.,  2004,  p.  3).  It  is  “akin  to  a  farmer  \A/ho 
\A/ishes  only  to  harvest  and  not  to  sow”  (Frosch,  1996,  p.  22).  To  sow 
the  seeds  of  technology,  and  to  increase  the  opportunity  for  greater 
capability  gains,  the  United  States  must  focus  more  effort  on  the 
earliest  stages  of  the  product  development  cycle,  researching  and 
experimenting  \A/ith  ne\A/  and  innovative  technologies.  “If  we  do  not 
invest  heavily  and  \A/isely  in  rebuilding  these  two  core  strengths”— 
S&Es  and  basic  R&D— “America  \A/ill  be  incapable  of  maintaining 
its  global  position  long  into  the  21st  century”  (U.S.  Commission  on 
National  Security,  2001,  p.  ix). 

Scientists  and  Engineers 

The  productive  po\A/er  of  the  U.S.  economy  and  its  national 
security  lies  primarily  \A/ith  its  people.  The  Office  of  Management  and 
Budget  (OMB)  estimates  that  privately  o\A/ned  capital  in  the  United 
States  is  \A/orth  $13  trillion,  \A/hile  its  human,  intellectual  capital  is 
\A/orth  $48  trillion  (OMB,  2007,  p.  195).  According  to  Alan  Greenspan, 
“If  we  are  to  remain  preeminent  in  transforming  kno\A/ledge  into 
economic  value,  the  U.S.  system  of  higher  education  must  remain 
the  \A/orld’s  leader  in  generating  scientific  and  technological  break¬ 
throughs  and  in  preparing  \A/orkers  to  meet  the  evolving  demands 
for  skilled  labor”  (Greenspan,  2000,  p.  4).  But  this  system  is  being 
challenged  from  abroad. 

Foreign  students,  particularly  Asian  students,  are  less  likely  to 
study  in  the  United  States  for  several  reasons.  First,  foreign  coun¬ 
tries  are  gro\A/ing  their  o\A/n  higher  education  capabilities.  From  1994 
to  1998,  the  number  of  Chinese,  South  Korean,  and  Tai\A/anese  doc¬ 
toral  students  at  U.S.  universities  dropped  by  19  percent,  \A/hile  their 
enrollment  at  institutions  in  their  native  countries  doubled  (Task 
Force,  2005,  p.  5).  In  2006,  five  Chinese  universities  ranked  in  the 
top  100  universities  for  science,  \A/ith  Peking  University  ranking  12th 
(Berry  &  Loeb,  2008,  p.  7).  Second,  foreign  countries  are  developing 
their  o\A/n  high-tech  industries  and  research  capacity.  As  a  result, 
increasing  numbers  of  U.S. -educated  doctoral  S&E  graduates  are 
returning  to  their  native  countries  to  pursue  research  opportunities. 
And  finally,  tighter  visa  restrictions  post-9/11  deter  foreign  students 
from  studying  in  the  United  States.  The  cap  on  H-1B  visas  for  high- 
skilled  specialties  decreased  from  115,000  in  2000  to  only  65,000  in 
2007  (Bordoff,  Deich,  Kahane,  &  Orszag,  2006,  p.  6).  Due  to  security 
restraints,  the  United  States  can  no  longer  rely  on  a  steady  influx 
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of  foreign  S&E  talent  to  supplement  innovation,  and  must  produce 
more  homegro\A/n  STEM  talent  to  maintain  its  economic  and  national 
security  edge. 

Since  1980,  S&E  positions  in  the  United  States  have  gro\A/n  by 
five  times  the  rate  of  other  professions;  ho\A/ever,  the  number  of  S&E 
degrees  earned  by  U.S.  citizens  is  decreasing  (Task  Force,  2005, 
p.  5).  This  is  especially  critical  to  DoD  laboratories  and  agencies 
like  the  National  Security  Agency,  \A/here  U.S.  citizenship  is  a  secu¬ 
rity  requirement  (Bordoff  et  al.,  2006,  p.  6).  Additionally,  the  time 
and  cost  to  pursue  S&E  graduate  degrees  have  increased  \A/hile  the 
compensation  in  S&E  fields  has  declined  relative  to  other  high-level 
occupations  (Freeman,  2005,  p.  10).  These  trends  clearly  signal  the 
need  to  create  incentives— such  as  higher  \A/ages,  fello\A/ships,  and 
employment  guarantees— to  maintain  the  pipeline  of  quality  S&E 
talent  that  our  nation’s  economy  and  national  security  structure 
sorely  need.  Unless  more  U.S.  students  choose  S&E  fields,  the  U.S. 
public  and  private  innovation  sectors  \A/ill  experience  a  significant 
“brain  drain.” 

The  DoD  is  taking  action  to  avoid  this  “brain  drain”  through 
NDEP.  The  objective  of  NDEP  is  to  bring  more  S&Es  into  the  national 
security  enterprise  by  supporting  local  educational  initiatives. 
NDEP  has  an  aggressive  congressional  mandate  to  a\A/ard  1,000 
innovative  scholarships  by  2013;  to  demonstrate  DoD’s  involvement 
in  K-12  education  programs;  and  to  a\A/ard  50  five-year  research 
fello\A/ships  by  2013.  NDEP’s  primary  focus  is  on  middle  school 
students  that  are  “at  a  game-changing  age  \A/here  they  \A/ill  need 
to  embrace  math  and  science,  or  likely  vanish  as  potential  STEM 
employees”;  ho\A/ever,  NDEP  also  focuses  on  university  students 
through  its  Science,  Mathematics,  and  Research  for  Transforma¬ 
tion  (SMART)  program.  SMART  funds  U.S.  S&E  students’  education 
costs  in  exchange  for  a  1-year  payback  in  a  DoD  laboratory  for  each 
year  of  educational  support.  While  NDEP  programs  have  sho\A/n 
success,  more  has  to  be  done  at  a  national  level  because  the  DoD 
has  83,000  S&Es  (70  percent  engineers)  and  replenishing  this 
resource  does  not  occur  overnight  (NDEP,  2009b,  pp.  1,  3).  The 
challenge  is  even  greater  in  industry. 

Industry  has  been  \A/orking  this  issue  for  some  time,  but  is  still 
struggling  to  hire  the  talent  it  needs.  For  example,  13  percent  of 
the  overall  aerospace  and  defense  \A/orkforce  is  qualified  for  retire¬ 
ment,  and  \A/ithin  10  years  this  figure  \A/ill  gro\A/  to  50  percent.  Of 
the  70,000  engineering  bachelor’s  degrees  aw/arded  in  the  United 
States  annually,  most  disciplines  are  not  in  high  demand  by  DoD 
contractors  (AIA,  2008,  p.  3).  Industry’s  viability  depends  on  a 
skilled  \A/orkforce,  so  industry  is  seizing  o\A/nership  of  the  issue. 
Lockheed  Martin  (LM),  the  top  recruiter  of  ne\A/  engineers  (5  percent 
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of  all  undergraduates  In  Its  nnajors  of  Interest),  Is  particularly  con¬ 
cerned  because  70  percent  of  Its  workforce  Is  over  40  (McPherson, 
2008a,  slide  5).  Through  Its  Engineers  In  the  Classroonn  progrann, 
LM  Is  building  school  partnerships  to  create  a  pipeline  of  future 
S&E  employees.  From  high  school  down  to  elementary  school,  LM 
engineers  are  participating  In  curriculum  development,  teacher 
training,  and  science  and  mathematics  extracurricular  activities  with 
the  objective  of  building  excitement  and  enthusiasm  for  science, 
math,  and  engineering  among  America’s  youth  (McPherson,  2008a, 
slides  7-10).  With  an  aerospace  and  defense  workforce  that  Is  half 
Its  size  at  the  end  of  the  Cold  War,  efforts  like  LM’s  Engineers  In 
the  Classroom  need  to  expand  In  size  and  numbers,  because  It  can 
take  22-25  years  to  grow  an  experienced  engineer  from  entry-level 
talent.  Meanwhile,  the  experienced  workforce  Is  retiring  at  accelerat¬ 
ing  rates  (McPherson,  2008b).  While  not  as  severe,  the  same  Issues 
apply  to  commercial  Industry.  A  possible  source  of  Increased  S&E 
talent  Is  women  and  minorities. 
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A  POSSIBLE  SOURCE  OE  INCREASED  S&E 
TALENT  IS  WOMEN  AND  MINORITIES. 
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As  hard  as  It  Is  to  attract  young  Americans  to  pursue  STEM,  It 
Is  even  harder  to  attract  women  and  minorities.  “The  proportions 
of  women,  blacks,  and  HIspanIcs  In  S&E  occupations  have  contin¬ 
ued  to  grow  over  time,  but  are  still  less  than  their  proportions  of 
the  population”  (NSB,  2008,  pp.  3-6).  While  women  make  up  46 
percent  of  the  overall  U.S.  workforce,  they  are  significantly  under¬ 
represented  in  the  S&E  professions  (Marshall  et  al.,  2004,  p.  4). 
Similarly,  “African  Americans  and  HIspanIcs  combined  make  up 
25  percent  of  the  U.S.  population,  but  account  for  only  11  percent 
of  the  engineering  bachelor’s  degrees  awarded  to  U.S.  students” 
(NDEP,  2009a,  p.  6).  Women  make  up  48.6  percent  of  the  college- 
degreed  workforce,  but  only  24.7  percent  of  the  S&E  workforce. 
African  Americans  constitute  7.4  percent  of  the  degreed  workforce 
and  only  6.9  percent  of  the  S&E  workforce;  while  HIspanIcs,  the 
largest  growing  population  in  the  United  States,  only  constitute 
4.3  percent  of  the  college-educated  workforce  and  3.2  percent  of 
the  S&E  workforce  (Marshall  et  al.,  2004,  p.  4).  The  significance 
of  these  figures  is  magnified  because  the  majority  of  women  and 
minority  S&Es  are  relatively  young;  therefore,  enticing  more  into 
the  S&E  professions  could  significantly  help  with  America’s  “brain 
drain”  of  S&E  talent  (Marshall  et  al.,  2004,  p.  4).  The  basic  R&D 
“budget  drain”  is  just  as  impacting. 
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Basic  Research  and  Development 

As  changes  in  this  century’s  threat  environment  create 
strategic  chaiienges—irreguiar  warfare,  weapons  of  mass 
destruction,  disruptive  technoiogies—this  request  pieces 
greater  emphasis  on  basic  research,  which  in  recent  years 
has  not  kept  pace  with  other  parts  of  the  budget. 

-  Robert  M.  Gates,  Secretary  of  Defense 

Secretary  Gates’  emphasis  on  basic  R&D  in  his  Fiscai  Year  2009 
Posture  Statement  is  encouraging;  however,  much  more  is  required 
at  a  nationai  ievei  (Rees,  2008a,  siide  7).  In  a  giobai  comparison 
of  the  basic  research  share  of  total  R&D  expenditures,  the  United 
States  ranks  16th  (NSB,  2008,  Figures  4-20,  4-41).  DoD’s  $271  million 
basic  R&D  increase  in  the  Fiscal  Year  2009  budget  is  a  step  in  the 
right  direction;  however,  it  may  do  little  to  overcome  years  of  declin¬ 
ing  and  flat  budgets  (Rees,  2008a,  slide  17).  In  the  post-9/11  world, 
where  security  and  economic  threats  can  appear  from  anywhere, 
a  diverse  and  vibrant  national  basic  R&D  program  is  a  necessity  for 
economic  and  national  security. 

Direct  contribution  of  R&D  investment  to  economic  growth  in 
real  Gross  Domestic  Product  (GDP)  was  6.7  percent  during  1995- 
2002,  up  from  4.3  percent  during  1974-1994  (Bureau  of  Economic 
Analysis,  2006).  Although  R&D  contributes  significantly  to  economic 
growth,  the  private  sector  invests  less  in  basic  R&D  than  is  justified 
by  societal  benefits,  because  private  innovators  receive  only  a  small 
fraction  of  the  benefits  their  inventions  generate.  Surveys  show  that 
private  return  on  investment  (ROI)  of  basic  R&D  typically  ranges 
between  7-15  percent,  while  the  social  benefit  ranges  between 
30-50  percent  (Popp,  2004).  While  society  benefits  tremendously, 
private  industry’s  incentive  to  conduct  basic  R&D  is  low  so  industry 
concentrates  on  short-term,  incremental  R&D,  similar  to  DoD.  This  is 
a  significant  innovation  loss,  because  the  private  sector  accounts  for 
nearly  two-thirds  of  total  R&D  (Bordoff  et  al.,  2006,  p.  2).  Because 
the  private  sector  invests  less  than  it  could  in  basic  R&D,  the  United 
States  is  not  realizing  its  full  economic,  or  for  that  matter,  national 
security  potential.  The  world’s  fastest  growing  economies  are  on 
track  to  catch  up  to  the  U.S.  basic  R&D  investment.  China,  South 
Korea,  and  Taiwan  increased  their  R&D  investment  by  140  percent 
from  1995-2001,  while  the  United  States  increased  its  investment 
by  only  34  percent  (Task  Force,  2005,  p.  9).  To  compensate  for  this 
shortage  in  private  basic  R&D,  the  United  States  must  increase  its 
public  investment  to  maintain  its  innovative  edge. 

“Much  of  the  strength  of  the  United  States  is  attributable  to 
its  technological  prowess,  much  of  which  developed  out  of  gov- 
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FIGURE  5.  DECREASING  TREND  IN  RESEARCH  AND 
DEVELOPMENT  (R&D)  AS  PERCENT  OF  GROSS  DOMESTIC 
PRODUCT,  FY  1976-2009* 
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*  FY  2009  figures  are  the  latest  American  Association  for  the  Advancement  of  Science  (AAAS)  estimates 
of  FY  2009  request.  R&D  includes  conduct  of  R&D  and  R&D  facilities.  Data  to  1994  are  obligations  from 
the  National  Science  Foundation  Federal  Funds  Survey.  The  2009  Gross  Domestic  Product  figures  are 
from  the  United  States  Government,  Office  of  Management  and  Budget. 


Note.  Adapted  from  AAAS  analyses  of  R&D  in  annual  AAAS  R&D  reports. 


ernment-funded  science  research”  (NDEP,  2009a,  p.  3).  Publicly 
funded  basic  R&D  typically  yields  a  high  ROI  of  30  percent  or 
more  (Mansfield,  1991,  p.  3).  However,  “federal  basic  research  in 
the  physical  sciences  and  engineering  is  flat  or  has  declined  as 
a  percentage  of  GDP  over  the  past  30  years”  (Figure  5)  (NDEP, 
2009a,  p.  3;  Advancing  Science,  Serving  Society  [ASSS],  2009, 
Figure  1).  Yet  in  times  of  crisis,  public  basic  R&D  has  paid  off.  Basic 
R&D— conducted  by  the  NSF,  the  National  Aeronautics  and  Space 
Administration,  and  DoD  for  example— has  provided  innovations 
with  huge  societal  payoffs,  such  as  the  World  Wide  Web,  portable 
communications,  the  Internet,  computer  graphics,  and  broadband 
capabilities  to  name  just  a  few,  some  of  which  have  led  to  multibil- 
lion-dollar  industries  (National  Research  Council,  2003).  The  range 
of  potential  security  and  economic  threats  is  increasing;  therefore, 
the  United  States  must  increase  its  investment  in  basic  R&D  across 
the  board.  While  DoD  has  been  and  is  currently  the  largest  bene¬ 
factor  and  contributor  to  public  basic  research,  it  is  losing  ground 
fast  (Rees,  2008b,  p.  iv). 

Recent  trends  raise  questions  about  the  U.S.  public  funding  of 
basic  R&D.  First,  basic  R&D  has  shifted  from  long-term,  blue-sky 
research,  which  will  most  likely  yield  significant  technological  break- 
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FIGURE  6.  INNOVATION  CURVE:  BASIC  R&D  PROVIDES  GREATER 
CAPABILITY  GAINS  AT  LESS  COST 


Cost,  $ 


FIGURE  7.  FEDERAL  OBLIGATIONS  FOR  R&D  BY  AGENCY  AND 
TYPE  OF  R&D;  FISCAL  YEAR  2007 
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Note.  DoD  =  Department  of  Defense;  DOE  =  Department  of  Energy;  DHS  =  Department  of 
Homeland  Security;  HHS  =  Department  of  Health  and  Human  Services;  NASA  =  National 
Aeronautics  and  Space  Administration;  NSF  =  National  Science  Foundation;  USDA  =  U.S. 
Department  of  Agriculture.  Detail  may  not  add  to  total  because  of  rounding.  Adapted 
from  Science  and  Engineering  indicators  2008,  by  National  Science  Board,  2008, 
Appendix,  Table  4-30;  and  National  Science  Foundation,  Division  of  Science  Resources 
Statistics,  Federal  Funds  for  Research  and  Development;  Fiscal  Years  2005,  2006,  and 
2007.  Due  to  rounding  issues  in  the  source  from  which  the  data  were  adapted,  not  all 
percentages  add  up  to  TOO  percent. 
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throughs  to  R&D  designed  to  reach  more  specific  findings  in  shorter 
time  horizons  and  at  greater  cost,  such  as  appiied  and  develop- 
mentai  R&D  (Figure  6).  Of  note  is  the  Defense  Advanced  Research 
Project  Agency’s  (DARPA)  shortening  of  go/no-go  revievt/s  for 
projects  that  are  from  projects  of  12-  to  18-month  intervais  versus 
prior  36-month  intervais.  This  type  of  short-term  focus  hamstrings 
researchers  and  reduces  the  possibility  of  groundbreaking  innova¬ 
tions  like  the  Internet,  global  positioning  technology,  and  Stealth 
that  DARPA-funded,  blue-sky  research  has  produced  in  the  past 
(Bordoff  et  al.,  2006,  p.  9).  Similarly,  the  Services  own  80  percent 
of  the  defense  basic  R&D;  ho\A/ever,  they  are  primarily  interested 
in  mission-focused  research,  not  blue-sky  research  (Rees,  2008a, 
slide  22).  Figure  7  sho\A/s  the  federal  R&D  obligations  for  Fiscal  Year 
2007.  What  is  most  telling  is  DoD’s  lack  of  interest  in  basic  R&D— 
only  2  percent  (NSB,  2008,  p.  4-25,  Figure  4-6).  If  DoD  v\/ants  to 

<><>00<><><><><X><X>0<XX><X>0<X><><X><>X>0<>0<X><><>^<X><>0<>000<X>0<>000<>0<X><>0<XXX>0<X><X><><X>0<X><^^ 

IF  DOD  WANTS  TO  CONTINUE  ITS  LONG-HELD 
STRATEGY  OF  “QUALITY  OVER  QUANTITY”  VIA 
HIGH-TECHNOLOGY  AND  MODERNIZATION,  IT  MUST 
CONTINUE  TO  EXPAND  ITS  BASIC  R&D  INVESTMENT. 
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continue  its  long-held  strategy  of  “quality  over  quantity”  via  high- 
technology  and  modernization,  it  must  continue  to  expand  its  basic 
R&D  investment.  Second,  the  public  basic  R&D  portfolio  reflects 
a  gro\A/ing  imbalance.  Betw/een  1995  and  2005,  biomedical  basic 
R&D  increased  by  115  percent— four  times  the  rate  of  increase  in 
basic  R&D  in  the  physical  sciences,  mathematics,  and  engineering, 
\A/hich  are  the  disciplines  most  applicable  to  DoD  initiatives  (AAAS, 
2006).  In  the  $789  billion  American  Recovery  and  Reinvestment 
Act  signed  into  la\A/  on  February  17,  2009,  $21.5  billion  is  for  federal 
R&D  \A/ith  the  majority— $10.4  billion  or  48.4  percent— going  to  the 
National  Institutes  of  Health  (NIH)  for  medical  research,  of  \A/hich 
$6.5  billion  alone  is  for  biomedical  R&D.  The  next  largest  share  at 
$3  billion— a  factor  of  3.5  times  less  than  the  NIH  share— \A/ent  to 
the  NSF  (Figure  8)  (AAAS,  2006).  While  advances  in  medicine  are 
\A/orthy,  limiting  research— particularly  research  in  the  physical  and 
engineering  sciences  that  apply  to  a  broad  array  of  scientific  fields, 
including  biomedical— limits  innovation  potential  (National  Research 
Council,  2001).  And  finally,  basic  R&D  does  more  than  just  gener¬ 
ate  ne\A/  discoveries  and  know/ledge;  it  also  prevents  technological 
surprise,  educates  S&Es  so  that  they  can  be  more  effective,  and 
sustains  the  human  talent  and  research  infrastructure  so  critical  to 
national  security  and  economic  gro\A/th  (Rees,  2008b,  p.  2).  To  stop 
and  reverse  this  innovation  implosion,  ne\A/  policies  are  needed  to 
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FIGURE  8.  2009  ECONOMIC  RECOVERY  BILL  FEDERAL  R&D 
FUNDING-HOUSE,  SENATE,  AND  FINAL  BILLS 

12 


Note.  NIH  =  National  Institutes  of  Health;  NSF  =  National  Science  Foundation;  DOE  = 
Department  of  Energy;  NIST  =  National  Institute  of  Standards  and  Technology;  NASA 
=  National  Aeronautics  and  Space  Administration.  Adapted  from  Final  Stimulus  Bill 
Provides  $21.5  Billion  for  Federal  R&D,  by  American  Association  for  the  Advancement 
of  Science,  2009. 


increase  public  and  private  R&D  as  \A/ell  as  incentives  to  increase  the 
number  of  quality  American-born  S&Es. 

Policy  Recommendations 

U.S.  technological  leadership  requires  effective  government 
policies  to  keep  the  nation  at  the  leading  edge  of  the  scientific  fron¬ 
tier.  “The  attack  [September  11,  2001]  \A/as  sort  of  like  \A/hen  Sputnik 
\A/ent  up  and  created  the  National  Defense  Authorization  Act  in 
1958”;  ho\A/ever,  Americans  need  to  once  again  find  the  excitement 
and  urgency  of  50  years  ago  that  led  to  technological  achievements 
such  as  the  Apollo  moon  landings  (NDEP,  2009a,  p.  2).  Time  is  of  the 
essence  because  the  Apollo  generation  is  ripe  for  retirement.  U.S. 
leadership  should  create  and  pass  a  National  Security  Education  Act 
for  the  21st  Century  to  provide  a  strategic  frame\A/ork  for  national 
security  and  economic  policies.  The  follo\A/ing  policies,  \A/hile  not  an 
exhaustive  list,  v\/ould  be  a  step  in  the  right  direction. 

National  Innovation  Policy  Recommendation 

Simply  funding  more  basic  research  and  educating  more  S&Es 
is  not  enough.  The  United  States  must  create  national  innova¬ 
tion  policies  to  provide  focus,  to  avoid  excessive  duplication  \A/ith 
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limited  research  dollars,  to  promote  the  diffusion  of  innovative 
ideas  across  private  and  public  lines,  to  advocate  for  innovative 
projects,  to  ensure  a  continuous  supply  of  quality  S&Es  and  basic 
R&D  resources,  and  to  tie  innovation  to  U.S.  economic  and  national 
security.  In  addition,  the  developer,  o\A/ner,  and  executor  of  innova¬ 
tion  policy  should  be  the  Office  of  Science  and  Technology  Policy 
headed  by  the  Science  Advisor  to  the  President,  thus  providing 
clout  to  innovation  policy. 

S&E  Policy  Recommendations 

Increase  the  number  and  value  of  S&E  graduate  research  fellowships 
(GRFs).  At  a  minimum,  the  NSF  should  triple  its  GRFs  to  restore  the 
ratio  of  GRFs  to  undergraduate  engineering  degrees  to  the  ratio 
that  existed  in  the  early  1960s,  follo\A/ing  Sputnik  (Freeman,  2006, 
pp.  2-3).  Increasing  GRFs  will  incentivize  the  most  talented  S&E 
students  to  continue  on  to  graduate  work  versus  pursuing  more 
lucrative  fields  (Bordoff  et  al.,  2006,  p.  7). 

Continue  to  attract  the  best  and  brightest  S&Es  from  abroad.  Highly 
skilled  immigrant  S&Es  contribute  significantly  to  U.S.  economic 
growth.  For  example,  a  third  of  all  businesses  founded  in  the  Silicon 
Valley  in  the  1990s  were  started  by  foreign-born  S&E  entrepreneurs. 
The  Russian-born  Sergei  Brin  started  Google;  and  eBay  was  started 
by  Pierre  Omidyar,  an  Iranian  born  in  Paris.  Also,  one-half  of  U.S. 
Nobel  laureates  in  science  are  foreign-born  (Bordoff  et  al.,  2006, 
p.  7).  And  finally,  foreign-born  competition  will  drive  U.S. -born  S&Es 
to  achieve  greater  educational  heights  and  innovation  to  compete. 

Increase  the  H-IB  visa  caps  to  pre-9/11  levels.  “The  U.S.  is  failing  to 
take  full  advantage  of  the  global  talent  pool”  (Bordoff  et  al.,  2006, 
p.  8).  The  number  of  international  S&E  students  in  U.S.  graduate 
programs  declined  by  20  percent  between  2001  and  2004  (Bordoff 
et  al.,  2006,  p.  8).  The  United  States  must  reverse  this  trend  to 
maintain  its  innovative  edge. 

Improve  STEM  education.  By  developing  programs  that  demonstrate 
the  practical  uses  of  math  and  science,  the  government  can  generate 
interest  in  STEM  careers  and  support  students  interested  in  these 
programs  through  government-funded  fellowship,  thus  providing  a 
steady  stream  of  S&E  talent. 

Basic  R&D  Policy  Recommendations 

Increase  public  basic  R&D  resources  (funding  and  facilities)  and  apply 
them  based  on  overall  effectiveness.  This  will  help  balance  basic  R&D 
investment  over  all  scientific  fields,  particularly  the  physical  sciences. 
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mathematics,  and  engineering  that  have  broad  appiications,  to 
avoid  overinvestment  in  particuiar  areas  such  as  biomedicai,  at  the 
expense  of  others. 

Incentivize  private  basic  R&D  through  R&D  tax  credits.  Tax  credits 
\A/ouid  incentivize  private  firms  to  conduct  basic  R&D  in  areas  that 
reflect  public  interest,  in  addition  to  increasing  innovation  and 
spin-off  commercial  opportunities  that  increase  economic  gro\A/th 
(Atkinson  &  Wial,  2008,  pp.  8, 11). 

Employ  prizes  when  applicable.  Prizes,  particularly  in  the  multimillion- 
dollar  range,  could  entice  researchers  that  \A/ould  other\A/ise  not 
do  business  \A/ith  the  government  due  to  bureaucratic  red  tape. 
In  addition  to  bringing  in  fresh  ideas,  prize  strategies  increase 
the  resources  brought  to  bear  on  a  problem  because  research 
teams  apply  their  own  funds  in  the  hope  of  \A/inning  (Bordoff  et  al., 

2006,  p.  10). 

Regulate  intellectual  property  rights  such  that  innovation  is  maximized 
while  protecting  innovator's  rights.  The  number  of  patents  granted 
increased  from  a  rate  of  less  than  1  percent  a  year  from  1930  to  1982 
to  5.7  percent  a  year  from  1983  to  2002.  In  addition,  the  intellectual 
rigor  required  to  receive  a  patent  also  decreased  (Jaffe  &  Lerner, 
2004,  p.  25).  As  a  result,  excessive  and  inappropriate  patents  keep 
innovation  out  of  the  public  realm  (Nelson  &  Romer,  1996,  p.  19). 


Conclusions 

Innovation  is  more  important  to  the  U.S.  economy  and  national 
security  no\A/  than  in  the  past.  Since  World  War  II,  the  United  States 
has  been  the  leader  in  innovation;  ho\A/ever,  international  competi¬ 
tion  is  posing  a  gro\A/ing  challenge  to  U.S.  technological  supremacy. 
The  United  States  has  the  best  market  environment  in  the  \A/orld  to 
support  innovation,  but  arguably  \A/eak  innovation  policies.  Effec¬ 
tive  government  innovation  policies  are  critical  to  keeping  the 
nation  at  the  leading  edge  of  the  scientific  frontier.  “What  makes 
kno\A/ledge,  innovation,  and  technology  such  po\A/erful  drivers  of 
economic  gro\A/th  is  that,  unlike  capital  and  labor,  they  do  not  suf¬ 
fer  from  diminishing  returns”;  therefore,  America  must  rebuild  its 
foundation  of  competitiveness— its  supply  of  S&E  talent  and  basic 
R&D  resources— that  have  served  the  country  so  \A/ell  for  the  past 
50  years  (Atkinson  &  Wial,  2008,  p.  19).  The  challenges  are  real  and 
gro\A/ing,  so  kno\A/ledge  generation  and  innovation  must  become 
a  national  priority.  Sir  Isaac  Ne\A/ton  captured  this  continuation  of 
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innovation  best  \A/hen  he  said,  “If  I  have  seen  further,  it  is  by  standing 
on  the  shoulders  of  giants”  (Atkinson  &  Wial,  2008,  p.  11). 
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A  STATISTICAL  APPROACH 
TO  THE  DEVELOPMENT 
OF  PROGRESS  PLANS 
UTILIZING  BAYESIAN 
METHODS  AND 
EXPERT  JUDGMENT 


1  Tiffany  L.  Lewis,  Thomas  Mazzuchi, 
and  Shahram  Sarkani 

The  development  of  progress  plans  for  each  identified  tech¬ 
nical  performance  parameter  (TPP)  is  a  critical  element  of 
technical  performance  measurement.  The  measured  values 
of  TPPs  are  referred  to  as  technical  performance  measures 
(TPMs).  These  terms  are  used  interchangeably;  however,  TPMs 
more  directly  reflect  how  technical  progress  and  technical 
risk  are  measured  and  evaluated.  Progress  plans,  or  planned 
performance  profiles,  are  crucial  to  effective  risk  assessment; 
however,  methods  for  developing  these  plans  are  subjective 
in  nature,  have  no  statistical  basis  or  criteria  as  a  rule,  and 
are  not  sufficiently  addressed  In  literature.  The  methodology 
proposed  herein  for  progress  plan  development  will  involve 
the  elicitation  of  expert  judgments  to  formulate  probability 
distributions  that  reflect  the  expected  values/estimates  used 
to  establish  progress  plans. 


Keywords:  Technical  Performance  Parameter  (TPP), 
Technical  Performance  Measure  (TPM),  Planned 
Performance  Profile,  Bayesian  Methods,  Technical 
Risk 
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The  development  of  individual  progress  plans  for  each  identified 
technical  performance  measure  (TPM)  is  a  critical  element  of  techni¬ 
cal  performance  measurement;  ho\A/ever,  the  methods  for  developing 
these  plans  are  subjective  in  nature,  have  no  statistical  basis  or  crite¬ 
ria  as  a  rule,  and  are  not  sufficiently  addressed  in  literature.  This  step 
is  arguably  considered  the  most  critical  aspect  of  technical  perfor¬ 
mance  measurement  because  it  provides  the  basis  for  forecasting 
successful  product  development  or  failure;  ho\A/ever,  the  absence  of 
clearly  defined  processes  to  develop  planned  performance  profiles 
is  a  void  that  exists  and  \A/ill  be  specifically  addressed  in  this  article. 
Bayesian  Methods  and  the  elicitation  of  expert  judgments  to  formu¬ 
late  probability  distributions  that  reflect  the  estimated  values  \A/ill  be 
utilized  to  establish  the  performance  profiles. 

The  Practical  Soft\A/are  and  Systems  Measurement  (PSM)  pro¬ 
cess  presented  in  Roedler  and  Jones  (2005)  is  highly  flexible  and 
provides  the  foundation  for  the  execution  of  Technical  Performance 
Measurement.  This  process  has  been  adopted  by  the  International 
Council  of  Systems  Engineering  as  an  accepted  practice,  and 
tailoring  this  approach  for  a  typical  DoD  program  results  in  a  five- 
phase  process:  identification,  planning,  measurement,  revie\A/,  and 
reporting.  Each  phase  consists  of  multiple  process  steps,  and  the 
methodology  discussed  herein  occurs  during  planning.  Thus,  the 
latter  three  phases  are  not  addressed  or  discussed  in  this  study. 

Within  the  identification  phase,  the  customer  establishes  tech¬ 
nical  goals  and  requirements,  program  priorities  are  defined,  and 
TPMs  are  identified  to  support  program  goals  and  priorities  ideally 
traceable  to  the  lo\A/est  level  \A/ork  breakdo\A/n  structure  (WBS)  \A/ork 
package  elements. 

The  pianning  phase  begins  upon  completion  of  the  identifica¬ 
tion  phase.  During  the  planning  phase,  program  goals  are  allocated 
by  WBS  \A/ork  package  \A/ith  respect  to  budget,  schedule,  and  the 
expected  maturity  of  the  technology  under  revie\A/.  Additionally,  dur¬ 
ing  the  planning  phase,  planned  performance  profiles  and  tolerance 
bands  are  developed  for  each  TPM  identified,  initial  risk  assessments 
are  conducted,  and  a  technical  performance  baseline  is  developed. 

Why  a  Statistical  Approach  is  Needed 

Establishing  the  technical  baseline  and  individual  progress  plans 
for  each  identified  TPM  is  a  critical  element  of  technical  perfor¬ 
mance  measurement  and  is  the  means  by  \A/hich  technical  progress 
and  technical  risk  are  measured  and  evaluated.  This  occurs  during 
the  planning  phase  as  \A/ell  as  risk  assessment,  and  these  activities 
are  consistent  \A/ith  proposed  TPM  implementation  methodolo¬ 
gies  found  in  current  literature  such  as  Roedler  and  Jones  (2005). 
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Utilizing  Bayesian  Methods  and  Expert  Judgment 


However,  it  is  important  to  note  that  a  review  of  current  iiterature 
suggests  that  the  processes  for  the  impiementation  and  execution 
of  technicai  performance  measurement  on  acquisition  programs 
are  not  weii  defined.  Coieman,  Kuiick,  and  Pisano  (1996,  p.  6)  refer 
to  them  as  being  ad  hoc  to  a  significant  degree  due  to  the  iack  of 
formaiiy  estabiished  practices  and  processes  in  industry  as  weii  as 
in  the  DoD.  The  PSM  guidebook  entitied  Practical  Software  and 
Systems  Measurement:  A  Foundation  for  Objective  Project  Man¬ 
agement  (Baiiey  et  ai.,  2000,  pt.  4,  p.  4-18)  refers  to  the  process 
for  deveioping  baseiine  pians  as  “Estimation”  and  states  that  poor 
estimates  often  iead  to  faiied  projects  and  resuit  from  the  lack  of 
systematic  estimation  processes  as  well  as  other  contributing  fac¬ 
tors.  Plans  are  developed  largely  through  consensus  by  program 
management,  developers,  systems  engineers,  and  subject  matter 
experts  who  rely  on  knowledge  from  previous  experience,  historical 
data  if  available,  and  known  cost  and/or  schedule  baselines.  These 
progress  plans  are  crucial  to  effective  risk  assessment;  however,  the 
methods  for  developing  them  are  subjective  in  nature,  have  no  sta¬ 
tistical  basis  or  criteria  as  a  rule,  and  are  not  sufficiently  addressed 
in  literature.  This  lack  of  formal  processes  to  establish  performance 
profiles  predisposes  them  to  an  inherent  degree  of  error  and  uncer¬ 
tainty,  above  and  beyond  that  inherent,  due  to  immature  technical 
development,  which  translates  into  higher  risk.  Thus,  employing  a 
formal  methodology  utilizing  an  established  statistical  approach  will 
minimize  the  level  of  uncertainty,  thereby  reducing  risk  exposure. 


Bayesian  Framework 


The  methodology  proposed  in  this  article  utilizes  an  expert 
judgment  model  within  a  Bayesian  framework  for  the  more  complex 
case  of  continuous  probability  distributions.  The  most  general  form 
of  Bayes’  Theorem  applies  to  discrete  probability  distributions,  and 
relates  the  conditional  and  prior  probabilities  of  two  events  using 
the  following  equation. 


(1) 


P(A\B)  is  considered  to  be  the  conditional  probability  of  ^  given 
B;  P(B\A)  is  considered  to  be  the  likelihood  of  B  given  A,  P(A)  is  the 
prior  probability  of  ^  (i.e.,  no  information  regarding  B  is  considered), 
and  P(B)  is  the  prior  or  marginal  probability  of  B. 

The  expert  judgment  model  utilized  in  this  study  for  the  case 
of  continuous  distributions  was  originally  presented  by  Mosleh  and 
Apostolakis  (1986)  to  estimate  seismic  fragility  curves  (i.e.,  the  con- 
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ditional  probability  that  a  seismic  stress  such  as  wind  or  earthquake 
will  cause  equipment  to  fail)— an  application  that,  upon  evaluation, 
proved  to  have  similarities  regarding  the  use  of  expert  opinion  elici¬ 
tation  for  planned  performance  profile  development.  In  both  cases, 
the  general  Bayesian  framework  for  continuous  distributions  uses 
the  opinions  of  experts  as  “evidence,”  and  this  evidence  is  used  as 
input  to  the  decision  maker’s  state  of  knowledge  using  Bayes’  Theo¬ 
rem.  Bayes’  Theorem,  in  its  general  form  for  continuous  probability 
distributions,  follows: 

Pr(x  I  E)  =  k-‘L(E  I  x)PrJx)  (2) 

Pr^(x)  is  the  prior  state  of  knowledge  (prior  distribution)  regard¬ 
ing  the  unknown  quantity  x  prior  to  obtaining  opinions  from  the 
experts;  E  is  the  set  of  opinions  provided  by  experts  about  the  value 
of  x;  L(E  \x)  =  the  likelihood  the  evidence  E  is  true,  given  the  true 
value  of  the  unknown  quantity  is  x;  Pr(x  \  E)  is  the  decision  maker’s 
posterior  state  of  knowledge  (posterior  distribution)  about  the 
unknown  quantity  x  given  the  set  of  opinions  E  provided  by  experts 
and  a  normalization  factor  A:',  which  is  used  to  makePrfx  \E)  a  prob¬ 
ability  distribution.  Within  this  framework,  the  likelihood  function 
can  be  equated  to  the  accuracy  level  of  the  expert's  estimate. 
Bayes’  Theorem  can  also  be  written  as  Equation  3  below  where  (a) 
consists  of  the  set  of  (m)  parameters  of  the  cumulative  (unknown) 
distribution  <l>(x\a). 

Pr  (a\E)  =  k'‘L(  E  \  a  )Pr^  (a)  (3) 

One  unique  curve  0(  x  |  a )  is  specified  by  each  vector  (a).  Thus, 
the  average  of  the  infinite  number  of  distributions  Of  x  |  a )  is  defined 
by  Equation  4  below. 

0(x)=1^0(x|a  )Pr(  a\E  )  da  (4) 

Assumptions 

The  following  assumptions  apply  to  the  application  of  the 
Mosleh  and  Apostolakis  (1986)  model  to  the  estimation  of  seismic 
fragility  curves  and  also  apply  to  estimation  of  progress  plan  values 
depicted  in  this  illustration. 


No.1. 

The  unknown  distribution  being  estimated  beiongs  to  a  para¬ 
metric  famiiy  of  distributions.  This  assumption  simplifies  the 
construction  of  the  likelihood  functions.  As  a  result,  the  challenge 
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of  estimating  the  unknown  distribution  is  reduced  to  the  estimation 
of  its  parameters. 


No.  2. 

The  unknown  (posterior)  distribution  is  assumed  to  be  iognormai 
with  parameters  (&)  and  (oj).  These  two  parameters  are  aiiowed  to 
vary  and  describe  the  variability  of  the  distribution.  By  definition, 
the  minimum  possible  value  for  a  lognormal  distribution  is  zero,  the 
maximum  possible  value  is  +  oo,  and  the  parameter  values  (mean  and 
standard  deviation)  must  be  greater  than  zero.  These  criteria  also 
apply  to  the  TPM  estimates  used  to  establish  a  planned  performance 
profile.  Therefore,  the  lognormal  distribution  was  deemed  the  most 
applicable  distribution  to  use  given  the  information  available. 


No.  3. 

Experts  are  independent  and  wiii  be  providing  independentiy 
assessed  percentiies.  This  assumption  is  intended  to  simplify  the 
complexity  of  the  model’s  application  to  this  problem.  A  model  for 
the  case  of  dependent  experts  is  presented  by  Mosleh  and  Apos- 
tolakis  (1986);  however,  it  will  not  be  explored  here. 


No.  4. 

Standard  deviations  assigned  to  each  expert  reflect  the  deci¬ 
sion  maker’s  ievei  of  confidence  in  the  expert’s  abiiity  to  accurateiy 
estimate  each  percentiie.  If  available,  historical  data  reflecting  the 
planned  versus  actual  values  of  prior  predictions  should  be  used 
to  determine  the  “bias”  or  “percent  error”  to  associate  with  each 
expert’s  accuracy  and  standard  deviation. 


No.  5. 

The  percentiies  being  estimated  are  assumed  to  be  symmetric. 
Symmetric  percentiles  (10  percent,  50  percent,  and  90  percent)  have 
been  assumed  for  the  model  presented  by  Mosleh  and  Apostolakis 
(1986).  'Values  for  these  percentiles  are  estimated  by  each  expert. 


No.  6. 

The  standard  deviations  of  the  percentiies  estimated  by  each 
expert  are  independent  of  the  percentiies  themseives.  In  other 
words,  each  expert  is  assumed  to  have  the  same  level  of  accu¬ 
racy  estimating  each  percentile  regardless  of  the  percentile  itself. 
Therefore,  the  standard  deviations  assigned  to  each  expert  for  each 
percentile  estimated  would  be  the  same.  However,  there  is  evidence 
to  support  that  experts  are  more  likely  to  be  less  accurate  at  higher 
percentiles,  thus  reflecting  a  greater  level  of  uncertainty  at  these 
levels  (George  &  Mensing,  1981). 
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Statistical  Approach 

The  methodology  proposed  is  presented  to  convey  the  feasi¬ 
bility  of  using  an  expert  judgment  model  developed  to  estimate 
seismic  fragility  curves  to  develop  retrospective  progress  plans. 
The  development  of  progress  plans  is  considered  to  be  a  key  and 
critical  process  element  of  technical  performance  measurement. 

The  Cockpit-21  TPM  project  described  in  the  unpublished  \A/hite 
paper  by  Coleman  et  al.  (1996)  presents  a  suitable  example  and 
adequate  data  and  information  to  convey  ho\A/  the  Mosleh  and 
Apostolakis  (1986)  methodology  could  have  been  employed  to 
develop  the  planned  performance  profiles  for  each  TPM  identified. 
The  example  utilizes  the  model  to  formulate  individual  estimates  for 
unkno\A/n  distributions  (as  opposed  to  single  values)  based  on  esti¬ 
mates  for  multiple  percentiles  provided  by  experts.  The  estimated 
distributions  reflect  values  of  specific  TPMs  for  different  points  in 
time  that  have  been  defined  to  coincide  \A/ith  key  milestone  dates. 
The  follo\A/ing  process  steps  are  proposed  for  the  development  of 
progress  plans  and  are  based  on  the  seismic  fragility  curve  model. 

Step  1.  Identify  TPMs. 

Step  2.  Define  dates  and  milestones  for  the  progress  plans  of  each 
parameter. 

Step  3.  Identify  experts  that  \A/ill  participate  in  the  estimation 
process. 

Step  4.  Assign  \A/eights  to  the  experts  identified. 

Step  5.  Assign  standard  deviations  to  the  experts  identified  for  each 
percentile  estimated.  Determine  standard  deviations  for  experts 
\A/ith  unique  \A/eights. 

Step  6.  Experts  estimate  a  TPM  value  for  each  established  percentile 
(10  percent,  50  percent,  and  90  percent)  for  each  milestone  date. 
Step  7.  Evaluate  the  standard  deviation  (o)  of  the  lognormal  param¬ 
eters  (0)  and  (co). 

Step  8.  Evaluate  the  value  of  the  lognormal  parameters  (0)  and  (co). 
Step  9.  Evaluate  the  distribution  curve  for  each  estimate  using 
Bayes’  Theorem. 

Step  10.  Define  tolerance  bands  for  established  performance 
profiles. 

Step  11.  Plot  the  mean  values  for  each  curve  against  its  respective 
milestone  date  to  establish  the  progress  plan. 
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TABLE  1.  COCKPIT-21  TECHNICAL  PARAMETERS  AND  WEIGHTS 


Display  Electronics  Unit  (DEU)  Parameters  (50%) 

Level  1  Parameter 

wt. 

Level  2  Parameter 

Wt. 

1.  Head-Up  Display 

20% 

1.1 

HUD  Design  and 

30% 

(HUD)  Module  Status 

1.2 

Code  Status 

HUD  Full 

Qualification  Test 

(FQT)  Status 

70% 

2.  Manpower 

20% 

3.  Multifunction  Display 

20% 

3.1 

MFD  Design  and 

30% 

(MFD)  Module  Status 

Code  Status 

3.2 

MFD  FQT  Status 

70% 

4.  Requirements 

20% 

Volatility 

5.  Software  Problem 

20% 

5.1 

Software  Problem 

50% 

Reports 

5.2 

Reports  Closed 
Software  Problem 
Reports  Open 

50% 

Flight  Test  Problem  Reporting  Parameters  (50%) 

Level  1  Parameter 

wt. 

6.  Reports  Closed 

50% 

7.  Reports  Open 

50% 

Identify  Technical  Performance  Measures 


Once  goals  and  requirements  have  been  set,  TPMs  are  identified 
to  support  the  program  goals  and  priorities.  The  TPMs  identified 
are  assumed  to  have  some  notable  impact  to  program  costs,  critical 
path,  or  technical  risk.  T\a/o  technical  areas  representing  50  percent 
each  of  the  technical  performance  baseline  were  identified  for  the 
Cockpit-21  project.  These  areas  were  the  Display  Electronics  Unit 
(DEU)  soft\A/are  and  Flight  Test  Problem  Reporting.  Table  1  displays 
the  technical  parameters  and  sub-parameters  identified  for  DEU  and 
for  Flight  Test  Problem  Reporting.  Additionally,  the  \A/eight  of  each 
parameter  at  each  level  is  sho\A/n. 

Define  Dates  and  Milestones 
for  the  Progress  Plans 

The  TPM  milestone  assessment  dates  for  the  Cockpit-21  program 
ranged  from  June  1992  through  January  of  1995,  and  the  date  range 
for  each  parameter  varied  \A/ith  respect  to  its  scheduled  develop- 
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TABLE  2.  COCKPIT-21  PROGRESS  PLAN  EVALUATION  DATES 


Display  Electronics  Unit 

Start 

Finish 

#of 

(DEU)  Parameters 

Date 

Date 

Estimates 

1.1  HUD  Design  and  Code  Status 

June  '93 

Feb  '94 

9 

1.2  HUD  Full  Qualification  Test  (FQT)  Status 

June  '93 

Feb  '94 

9 

2.  Manpower 

June  '92 

Feb  '94 

21 

3.1  MFD  Design  and  Code  Status 

June  '93 

Feb  '94 

9 

3.2  MFD  Full  Qualification  Test  (FQT)  Status 

June  '93 

Feb  '94 

9 

4.  Requirements  Volatility 

Nov  '92 

Feb  '94 

16 

5.1  Software  Problem  Reports  Closed 

Nov  '92 

Feb  '94 

16 

5.2  Software  Problem  Reports  Open 

Oct  '92 

Feb  '94 

17 

Flight  Test  Problem 

Start 

Finish 

#of 

Reporting  Parameters 

Date 

Date 

Estimates 

6.  Reports  Closed 

Mar  '94 

Jan  '95 

11 

7.  Reports  Open 

Mar  '94 

Jan  '95 

11 

merit  within  the  project’s  iife  cycie.  Assessments  were  scheduied  at 
monthiy  intervais  for  each  parameter;  however,  Roedier  and  Jones 
(2005)  indicate  that  it  is  advisabie  to  utiiize  significant  miiestones 
and/or  design  events  to  estabiish  performance  profile  measure¬ 
ment  dates.  Coleman  et  al.  (1996)  also  states  that  dates  should  be 
based  on  events  during  the  development  cycle  as  opposed  to  a 
periodic  scheme.  Therefore,  this  would  be  the  preferred  method  for 
establishing  evaluation  dates  for  TPM  values  when  implementing 
technical  performance  measurement  on  future  projects.  The  range 
of  dates  and  number  of  estimates  associated  with  each  progress 
plan  developed  for  the  Cockpit-21  program  are  depicted  in  Table  2. 
Since  TPMs  are  tracked  at  the  lowest  level,  it  is  only  necessary  to 
develop  performance  profiles  for  the  lowest  level  parameter.  Thus, 
for  the  purpose  of  this  application,  progress  plans  are  not  developed 
for  level  2  parameters  that  have  level  3  subparameters;  those  level 
2  parameters  are  not  depicted  in  Table  2. 

Identify  Experts 

The  experts  involved  in  the  TPM  planning  process  on  a  DoD 
program  will  most  often  be  the  members  of  the  integrated  product 
teams  (IPTs)  established  to  test,  monitor,  and  evaluate  technical 
progress.  IPTs  will  primarily  consist  of  contractors/developers. 
However,  government  representatives  may  work  with  contractors  to 
share  responsibilities  pertaining  to  the  planning  process.  Members 
of  each  IPT  are  selected  based  on  their  knowledge  and  experience 
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regarding  previous  deveiopnnent  efforts  \A/ith  the  same  or  simiiar 
technoiogy  and  may  be  systems  engineers,  test  engineers,  research¬ 
ers,  or  anaiysts.  For  the  purpose  of  this  study,  the  IPT  members  are 
considered  to  be  the  experts,  and  the  IPT  ieads  are  considered  to 
be  more  kno\A/iedgeabie  than  the  other  members  of  the  team. 

The  question  of  ho\A/  many  experts  are  optimai  \A/as  addressed 
by  K.  Waiker  (personai  communication,  November  11,  2004).  She 
concluded  there  has  been  no  evidence  of  an  existing  argument  for 
an  ideally  statistically  based  sample  size.  Her  literature  revie\A/  of  38 
studies  revealed  90  percent  used  11  or  less  experts,  and  referenced 
Steve  Hora,  \A/ho  has  been  repeatedly  quoted  for  his  argument  of 
“three  and  seldom  more  than  six”  is  sufficient  (Hora,  2004,  p.  5;  K. 
Walker,  personal  communication,  November  11,  2004).  The  example 
presented  in  this  study  assumes  four  experts  are  participating  in  the 
progress  plan  development  process  as  members  of  an  IPT. 

Assign  Weights  to  the  Experts 

Weights  are  assigned  by  the  decision  maker  to  each  expert 
providing  an  estimate  as  a  means  to  place  more  or  less  value  on 
the  responses  of  experts  that  are  assumed  to  have  greater  or  lesser 
accuracy  \A/ith  respect  to  their  estimation  ability.  The  more  confi¬ 
dence  a  decision  maker  has  in  a  particular  expert’s  ability,  the  more 
\A/eight  the  expert’s  estimate  should  carry  \A/ith  respect  to  other 
experts.  The  \A/eights  of  all  experts  should  be  normalized  so  that 
they  sum  to  one.  If  available,  this  step  is  v\/here  it  \A/ould  be  appro¬ 
priate  to  revie\A/  historical  data  from  previous  projects  to  evaluate 
the  estimation  accuracy  of  experts  conducting  the  estimates  to 
determine  the  appropriate  \A/eighting  scheme.  Additionally,  the 
experience  level  of  each  expert  (or  IPT  member)  and  their  level  of 
kno\A/ledge  regarding  the  technical  area  being  evaluated  should  be 
considered  \A/hen  assigning  \A/eights.  The  assessment  by  the  deci¬ 
sion  maker  in  this  example  assumes  each  expert  is  equally  \A/eighted 
\A/ith  the  exception  of  the  IPT  lead.  The  lead  is  given  more  \A/eight  to 
compensate  for  additional  kno\A/ledge  and  experience  in  the  techni¬ 
cal  area  of  interest. 

Assign  Standard  Deviations  to  the  Experts 

standard  deviations  are  assigned  by  the  decision  maker  to 
each  percentile  estimated  by  each  expert.  Each  standard  deviation 
reflects  the  perceived  range  of  deviation  from  the  true  value  being 
estimated  at  that  percentile.  The  approach  to  assigning  standard 
deviations  is  similar  to  that  of  assigning  \A/eights  in  that  historical 
data  from  previous  projects  should  be  revie\A/ed,  if  available,  to 
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evaluate  the  true  estimation  accuracy  or  percentage  error  realized 
by  experts  \A/ith  prior  predictions.  This  should  ensure  the  proper 
calibration  of  the  experts  and  allo\A/  the  decision  maker  to  account 
for  overconfidence  of  the  experts  as  discussed  by  Hora  (2004),  and 
errors  of  estimation  as  discussed  by  Winkler  (1981).  In  the  case  of 
the  Cockpit-21  project,  data  from  previous  TPM  pilot  implementation 
projects  discussed  by  Coleman  et  al.  (1996),  such  as  the  Air  Deploy¬ 
able  Active  Receiver  (ADAR)  sonobuoy  development  program  and 
the  LAMPS  Block-ll  Upgrade  program,  \A/ould  serve  as  appropriate 
reference  materials  for  this  effort.  The  model  presented  in  this  study 
assumes  the  standard  deviations  of  all  experts  are  equal  \A/ith  the 
exception  of  the  IPT  lead,  \A/ho  has  a  greater  \A/eight  than  the  remain¬ 
ing  team  members  (i.e.,  \A/here  (J)  equals  the  Jth  percentile 

and  (K)  equals  VneKth  expert.  To  calculate  the  standard  deviation  of 
the  IPT  lead.  Equation  5a  must  be  used  \A/here  A^=  the  total  number 
of  experts,  =  the  standard  deviation  of  expert  (K),  and  w  =  the 
w/eight  of  expert  (K). 

N 

= -  (5a) 

K=I  Wj 

This  \A/ill  result  in  the  standard  deviation  of  the  IPT  lead  being 
less  than  the  remaining  IPT  members  due  to  the  lead  being  assigned 
a  heavier  \A/eight. 

Estimate  TPM  Values  by  Percentile 

The  methodology  proposed  here  requires  TPM  values  to  be 
estimated  by  each  expert  for  each  established  percentile  (10  per¬ 
cent,  50  percent,  and  90  percent)  by  milestone  date.  Meaning,  each 
expert  must  predict  the  true  value  that  has  a  10  percent  likelihood 
of  being  observed,  a  50  percent  likelihood  of  being  observed,  and 
a  90  percent  likelihood  of  being  observed  for  the  given  evaluation 
date.  Experts  formulate  estimates  based  on  the  information  they 
have  available  to  them  at  the  time,  and  this  information  may  include 
historical  and/or  current  test  data,  the  results  of  formal  functional 
analysis,  and  expert  opinion  based  on  kno\A/ledge  and  experience. 

Table  3  depicts  the  proposed  estimations  for  the  first  milestone 
date  of  the  first  technical  parameter  sho\A/n  in  Table  2  (Display  1.1, 
HUD  Design  and  Code  Status),  the  basis  for  the  response  provided, 
as  \A/ell  as  the  assigned  \A/eight  and  standard  deviation  for  each 
expert.  Equation  5a  \A/as  used  to  estimate  (o^)as  follo\A/s: 
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4  0-*  (lO)-* 

= - = - =416,7 


K 

.24 

'  2. 

• 

Vi 

1 

”4= 

1 

.28  (416.7) 

(5  b) 


TABLE  3.  HUD  DESIGN  AND  CODE  STATUS  EXPERT  OPINION  DATA 


Expert 

HUD  Design  and  Code  Status  -  June  1993 

Expert  Expert  std.  Percentiles  Basis  for 

wt.  dev.  (ok)  10%  50%  90%  Response 

1 

.24 

0.100 

22.0 

23.0  26.0 

test  data,  analytical 

methods 

2 

.24 

0.100 

21.5 

22.8  24.0 

test  data,  analytical 
methods,  expert 
opinion 

3 

.24 

0.100 

22.0 

23.5  24.5 

test  data,  analytical 

methods 

4 

.28 

0.093 

21.5 

24.0  25.0 

test  data,  analytical 
methods,  expert 
opinion 

Using  the  values  in  Table  2,  it  is  evident  that  each  expert  must 
provide  27  unique  estimates  (9  estimates  x  3  percentiles)  to  estab¬ 
lish  a  progress  plan  for  this  parameter;  and  a  total  of  12  estimates  (3 
percentiles  x  4  experts)  \A/ill  be  used  in  the  expert  judgment  model 
to  establish  the  unkno\A/n  distribution  for  each  individual  estimate. 
To  complete  the  expert  judgment  analysis  for  all  parameters  and  all 
evaluation  dates,  this  process  \A/ould  be  completed  for  each  param¬ 
eter  sho\A/n  in  Table  2,  resulting  in  each  expert  providing  378  unique 
estimates  (126  estimate  dates  x  3  percentiles  per  date). 

Evaluate  the  Standard  Deviation 
of  the  TPM  Values 

The  unkno\A/n  distribution  associated  \A/ith  each  estimate  is 
assumed  to  be  lognormal  \A/ith  parameters  (o)  and  (co).  The  Mosleh 
and  Apostolakis  (1986)  model  uses  Equations  6  and  7  to  evaluate 
the  values  of  the  standard  deviations  for  (0)  and  (co).  The  standard 
deviation  for  (0)  is  denoted  by  Equation  6. 
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(6) 


K=1 


reflects  the  standard  deviation  for  (0).  The  standard  devia¬ 
tion  for  (to)  is  denoted  by  Equation  7,  \A/here  reflects  the  standard 
deviation  of  (co)  and  =  1.285  fronn  tables  of  the  standard  normal 
distribution. 


-I 


N 


(7) 


K=] 


Evaluate  the  Value  of  the  Lognormal  Parameters 

The  unkno\A/n  distribution  associated  \A/ith  each  estimate  is 
assumed  to  be  lognormal  \A/ith  parameters  (0)  and  (co).  The  Mosleh 
and  Apostolakis  (1986)  model  uses  Equations  8  and  9  to  evaluate 
the  values  of  the  parameters  (0)  and  (co).  The  parameter  value  for  (0) 
is  denoted  by  Equation  8. 


N 


(8) 


K=J 


The  parameter  value  for  (m)  is  denoted  by  Equation  9. 


N 


=  i  On^90K-  fnxjj 


(9) 


Evaluate  Each  Unknown  Distribution 
Using  Bayes'  Theorem 


No\a/  that  \A/eights  and  standard  deviations  have  been  assigned 
to  the  experts,  values  have  been  estimated  for  each  percentile  by 
the  experts,  the  standard  deviations  of  the  parameter  values  have 
been  evaluated,  and  the  parameter  values  themselves  have  been 
evaluated,  all  requisite  information  is  available  to  evaluate  the  value 
of  the  unkno\A/n  (posterior)  distribution  for  the  parameter  depicted 
in  Table  3.  Using  the  Mosleh  and  Apostolakis  model  for  Bayes’  Theo¬ 
rem,  the  posterior  distribution  can  be  evaluated  using  Equation  10. 
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(10) 


Upon  evaluation  of  Equation  10  for  the  parameter  depicted  in 
Table  3,  and  assuming  the  same  experts  are  providing  all  estimates, 
steps  7  through  9  are  repeated  for  the  remaining  parameters  identi¬ 
fied  in  Table  2  to  complete  the  expert  judgment  analysis. 

Define  Tolerance  Bands  for  Performance  Profiles 

Follo\A/ing  completion  of  the  expert  judgment  analysis,  the  next 
step  is  to  define  tolerance  bands.  Tolerance  bands  reflect  the  allo\A/- 
able  range  of  variation  and  level  of  acceptable  risk  for  a  defined 
TPM  estimate  on  a  given  milestone  date.  They  alert  management 
that  actions  may  be  necessary  to  get  the  TPM  back  on  track.  Actual 
values  that  exceed  the  allo\A/able  range  (e.g.,  20  percent)  for  any 
TPM  estimate  denote  high  risk  (red)  and  \A/ill  trigger  management 
intervention.  Additionally,  allo\A/able  ranges  are  defined  for  lo\A/-risk 
items  (green),  and  medium-risk  items  (yello\A/)  as  \A/ell.  Typically,  the 
bands  depicted  on  a  performance  profile  represent  the  maximum 
allo\A/able  variation. 

Plot  the  Expected  Values  for  Each  Curve 

If  the  expected  values  defined  in  the  Cockpit-21  \A/hite  paper 
are  assumed  to  be  the  mean  expected  values  determined  using 
the  expert  judgment  methodology,  these  values,  along  \A/ith  the 
tolerance  bands  and  the  threshold  values,  \A/ould  establish  the 
performance  baseline  for  each  performance  measure  \A/hen  plot¬ 
ted  against  their  respective  milestone  dates.  The  Figure  depicts 
the  expected  values  used  to  define  the  individual  progress  plan 
for  HUD  Design  and  Code  and  the  corresponding  dates  for  each 
estimate.  Actual  values,  a  revised  plan,  and  the  threshold  value  are 
also  plotted  in  the  example  sho\A/n  for  comparison  \A/ith  the  baseline 
expected  values. 


Discussion 

This  article  describes  ho\A/  expert  judgment  can  be  utilized 
\A/ithin  a  Bayesian  frame\A/ork  to  develop  a  formal  statistical  model 
to  quantify  expert  opinions  as  probability  distributions  for  the 
purpose  of  establishing  TPM  progress  plans.  To  demonstrate  these 
attributes,  key  assumptions  \A/ere  made,  and  actual  TPM  data  \A/ere 
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FIGURE.  HUD  DESIGN  AND  CODE  PROGRESS  PLAN  EXAMPLE 
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used  to  illustrate  the  application  of  the  nnethodology.  Baseline 
progress  plans  are  crucial  to  effective  risk  assessment;  ho\A/ever, 
the  methods  for  developing  these  plans  for  each  parameter  are 
subjective  in  nature,  have  no  statistical  basis  or  criteria  as  a  rule, 
and  are  not  sufficiently  addressed  in  literature.  This  lack  of  formal 
processes  to  establish  performance  profiles  predisposes  them  to 
an  inherent  degree  of  error  and  uncertainty  (i.e.,  risk).  The  formal 
methodology  presented  in  this  article  offers  an  alternative  that  \A/ill 
arguably  produce  more  accurate  progress  plans  and  \A/ill  minimize 
the  level  of  uncertainty,  thereby  resulting  in  reduced  risk  exposure. 
This  proposed  methodology  provides  program  management  \A/ith 
another  decision-making  tool  that  can  be  used  to  strengthen  estab¬ 
lished  systems  engineering  processes. 
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ANALYSIS  OF 
GENERATION  Y 
WORKFORCE 
MOTIVATION  USING 
MULTIATTRIBUTE 
UTILITY  THEORY 

( Ian  N.  Bar  ford  and  Patrick  T  Hester 

This  article  explores  the  difference  in  assigned  levels  of  work¬ 
place  motivation  and  happiness  between  federal  government 
workforce  members  of  Generation  Y  versus  Generation  X  and 
Baby  Boomers.  Thirty  hypotheses  were  tested,  and  11  were 
found  to  be  statistically  significant.  Generation  Y  does  assign 
different  levels  of  importance  and  partially  assigns  different 
levels  of  happiness  to  the  five  motivational  factors  examined  in 
this  study:  responsibilities,  compensation,  work  environment, 
advancement  potential,  and  free  time.  Advancement  potential 
and  free  time  were  rated  the  highest  factors  when  compared 
to  Generation  X  and  Baby  Boomers.  Sample  size  was  small 
due  to  limited  availability  of  workforce  members.  This  study 
represents  the  first  attempt  to  explore  motivational  factors  for 
the  Generation  Y  workforce  within  the  federal  government. 


Keywords:  Baby  Boomers,  Generation  X, 
Generation  Y,  Motivational  Factors,  Utility 
Measurement 
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Introduction 

Researchers,  supervisors,  and  hunnan  resource  professionals 
have  long  struggled  \A/ith  perfecting  managennent  strategies  for 
ennployees.  The  three  nnost  prevalent  \A/orking  generations  currently 
are  Baby  Boomers,  Generation  X,  and  Generation  Y.  To  understand 
Generation  Y’s  employment  motivations  and  attitudes,  two  ideas 
must  be  discussed:  (a)  a  \A/orking  definition  of  generation,  and  (b)  an 
understanding  of  preceding  generations’  motivations  and  attitudes. 

Several  prevalent  definitions  of  “generation”  exist.  Kupper- 
schmidt  (2000)  defines  a  generation  as  an  identifiable  group,  or 
cohort,  \A/hich  shares  birth  years,  age,  location,  and  significant  life 
events  at  critical  developmental  stages.  Palese,  Pantali,  and  Saiani 
(2006)  categorize  generations  as  those  born  \A/ithin  the  same  his¬ 
torical  timeframe  and  culture.  Crumpacker  and  Crumpacker  (2007) 
add  that  birth  rate,  along  \A/ith  historical  events,  defines  each  gen¬ 
eration.  These  groups  develop  a  unique  pattern  of  behavior  based 
on  these  common  experiences  (Kupperschmidt,  2000). 

Further  exploration  of  literature  sho\A/s  that  two  common  ele¬ 
ments  distinguish  a  generation:  the  birth  rate  and  significant  life 
events  (Crumpacker  &  Crumpacker,  2007;  Kupperschmidt,  2000; 
Smola  &  Sutton,  2002;  Cennamo  &  Gardner,  2008;  Sayers,  2007). 
When  the  birth  rate  increases  and  remains  steady,  that  signifies 
the  beginning  of  a  ne\A/  generation.  When  the  birth  rate  of  a  ne\A/ly 
formed  generation  begins  to  decline,  that  marks  the  end  of  a 
generation  (Crumpacker  &  Crumpacker,  2007).  Each  generation 
has  its  o\A/n  set  of  significant  life  events.  Each  generation  shares 
the  same  experiences,  or  is  a\A/are  of  them,  as  they  advance  and 
mature  through  different  stages  of  life— although  not  every  per¬ 
son  in  a  generation  personally  experiences  these  defining  events 
(Crumpacker  &  Crumpacker,  2007).  Caution  is  given  to  stereotyp¬ 
ing  individuals  based  on  generational  values  and  characteristics 
(Weingarten,  2009). 

The  eldest  of  the  current  \A/orking  generations,  referred  to  as  the 
Baby  Boomers,  were  born  bet\A/een  the  years  1946  and  1964  (Egri 
&  Ralston,  2004;  Smola  &  Sutton,  2002;  Westerman  &  Yamamura, 
2007;  Dries,  Pepermans,  &  DeKerpel,  2008;  Crumpacker  &  Crump¬ 
acker,  2007;  Hubbard  &  Singh,  2009).  Baby  Boomers  experienced 
significant  life  events  that  shaped  their  values,  including  the  social 
revolution  of  the  1960s,  the  \A/omen’s  movement.  President  John  F. 
Kennedy/Martin  Luther  King  Jr./Senator  Robert  F.  Kennedy  assas¬ 
sinations,  U.S.  landing  on  the  moon,  the  substantial  role  of  television 
\A/ithin  society,  the  Vietnam  War,  the  Watergate  scandal,  and  high 
inflation  of  the  1980s  (Dries  et  al.,  2008;  Crumpacker  &  Crumpacker, 
2007;  Weingarten,  2009). 
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Baby  Boomers  are  classified  with  such  values  and  attributes  as 
team  orientation,  optimism  (Hess  &  Jepsen,  2009),  and  expecting 
the  best  from  life  (Smola  &  Sutton,  2002).  Prior  to  the  1980s,  this 
generation  knew  of  prosperity  and  fortunate  outcomes  (Kupper- 
schmidt,  2000)  being  the  center  of  their  parents’  world  (Crumpacker 
&  Crumpacker,  2007),  similar  to  the  prosperity  that  Generation  Y  has 
been  accustomed  to  (Shih  &  Allen,  2007).  During  the  recession  in 
the  1980s,  businesses  downsized  and  reorganized,  which  conveyed 
to  the  Baby  Boomers  that  a  lifetime  career  with  one  organization 
might  not  be  a  certainty  (Mirvis  &  Hall,  1994).  Because  of  this.  Baby 
Boomers  were  characterized  as  free  agents  in  the  workplace  (Kup- 
perschmidt,  2000),  described  by  Crumpacker  and  Crumpacker 
(2007)  as  highly  competitive  micromanagers,  irritated  by  lazy 
employees,  with  a  positive  demeanor  towards  professional  growth. 

The  middle  cohort  of  current  working  generations,  referred  to 
as  Generation  X,  was  born  between  the  years  1965  and  1979  (Egri 
&  Ralston,  2004;  Smola  &  Sutton,  2002;  Crumpacker  &  Crump¬ 
acker,  2007),  and  it  has  the  least  amount  of  people  of  the  three 
generations  under  review.  For  this  generation,  the  life  events  that 
had  a  profound  impact  were  the  Iranian  hostage  crisis,  Iran  Contra 
scandal,  introduction  of  HIV/AIDS  as  a  pandemic,  oral  contracep¬ 
tive  pills,  the  1973  oil  crisis,  the  impeachment  of  President  Richard 
M.  Nixon,  introduction  of  computers  and  the  Internet,  and  the  Cold 
War  (Dries  et  al.,  2008;  Crumpacker  &  Crumpacker,  2007;  Weingar- 
ten,  2009).  As  Generation  X  matured,  so  did  technology  (Cennamo 
&  Gardner,  2008). 

This  generation  grew  up  with  both  parents  in  the  workforce,  or 
in  a  divorced  household,  and  as  a  result,  many  were  latchkey  kids, 
becoming  independent  at  a  young  age  (Crumpacker  &  Crumpacker, 
2007;  Weingarten,  2009).  Smola  and  Sutton  (2002)  describe  this 
generation  as  experiencing  social  insecurity,  rapidly  changing  sur¬ 
roundings,  and  a  lack  of  solid  traditions.  Generation  X  carried  the 
trend  of  distancing  themselves  from  companies  just  as  the  Baby 
Boomers  did  (Dries  et  al.,  2008),  making  them  distrustful  of  orga¬ 
nizations  (Westerman  &  Yamamura,  2007).  Generation  X  entered 
the  workforce  competing  with  the  Baby  Boomers  for  jobs  during 
the  1980s’  recession,  which  made  many  of  these  individuals  cynical 
towards  the  older  generation  (Crumpacker  &  Crumpacker,  2007). 

The  newest  generation  to  enter  the  workforce  was  born  between 
the  years  1980  and  2000  (Weingarten,  2009;  Cennamo  &  Gardner, 
2008;  Sayers,  2007).  Although  authors  differ  as  to  when  Genera¬ 
tion  Y  either  begins  or  ends  (Smola  &  Sutton,  2002;  Kupperschmidt, 
2000;  Hess  &  Jepsen,  2009;  Westerman  &  Yamamura,  2006;  Crum¬ 
packer  &  Crumpacker,  2007;  Broadbridge,  Maxwell,  &  Ogden,  2007; 
Sayers,  2007),  prevalent  literature  agrees  on  Generation  Y  begin- 
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ning  in  1980  (Snnola  &  Sutton,  2002;  Weingarten,  2009;  Crumpacker 
&  Crumpacker,  2007;  Essinger,  2006)  and  ending  in  2000  (Ciark, 
2007).  Other  terms  associated  with  Generation  Y  are  “Miiienniais” 
(Howe  &  Strauss,  2000),  “Net  Generation”  (Shaw  &  Fairhurst,  2008), 
and  “Generation  Next”  (Loughiin  &  Bariing,  2001;  Zemke,  Raines,  & 
Fiiipczak,  2000;  Martin,  2005). 

The  momentous  events  that  Generation  Y  experienced  were  the 
faii  of  the  Beriin  Waii,  the  induction  of  music  teievision  (MTV)  into 
society,  Coiumbine  High  Schooi  shootings,  9/11  terrorist  attacks, 
more  frequent  naturai  disasters,  and  the  obesity  epidemic  (Dries  et 
ai.,  2008;  Crumpacker  &  Crumpacker,  2007).  Sujansky  (2002)  writes 
that  this  generation  has  seen  more  substantiai  iife-changing  events 
eariy  on  than  preceding  cohorts.  Possibiy  the  most  significant  dif¬ 
ference  this  generation  possesses  over  others  is  the  integration 
of  technoiogy  into  their  daiiy  iives  and  the  omnipresence  of  how 
technoiogy  has  aiways  been  in  their  worid  (Obiinger,  2003;  Martin, 
2005;  Weingarten,  2009).  Martin  (2005)  describes  Miiienniais  as 
independent,  confident,  and  seif-reiiant.  This  may  be  due  to  the 
extensive  protection  and  praise  given  to  them  throughout  their 
formative  years  (Crumpacker  &  Crumpacker,  2007). 

In  business.  Generation  Y  exhibits  the  propensity  for  working  in 
teams  whiie  being  coiiaborative,  resuits-oriented  individuais,  and 
having  an  ardor  for  pressure  (Shih  &  Aiien,  2007).  Unfortunateiy, 
Generation  Y  foliowed  their  two  previous  generations  and  have 
partitioned  themseives  away  from  organizations  (Dries  et  ai.,  2008), 
knowing  that  iifetime  empioyment  at  an  organization  is  very  uniikeiy. 
Generation  Y  expects  to  change  jobs  often  during  their  iifetime 
(Morton,  2002;  Kim,  Knight,  &  Crutsinger,  2009),  especiaiiy  if  their 
taients  are  underutiiized  (Kim  et  ai.,  2009;  Weingarten,  2009). 
Miiienniais  want  iifeiong  iearning  (Aich,  2000),  expect  on-the-job 
training  (Morton,  2002)  to  stay  marketabie  (Sayers,  2007;  Holden 
&  Harte,  2004;  King,  2003),  and  proactively  plan  their  own  careers 
and  professional  development  (Westerman  and  Yamamura,  2007; 
Kim  et  ai.,  2009;  Zemke  et  ai.,  2000). 

Generation  Y  aspires  for  a  work/life  balance  (Crumpacker  & 
Crumpacker,  2007;  Zemke  et  ai.,  2000)  to  achieve  professional 
satisfaction  and  personal  freedom  (Sayers,  2007).  Generation  Y 
is  almost  automatic  at  multitasking  with  technology  as  if  it’s  an 
extension  of  their  being  (Freifield,  2007;  Kofman  &  Eckler,  2005; 
Rowh,  2007;  Loughiin  and  Barling,  2001),  and  may  change  a  job 
task  considerably  to  create  a  more  appealing  outcome  (Wrzesn- 
iewski  &  Dutton,  2001).  They  need  clear  directions  and  management 
assistance  for  tasks,  while  expecting  freedom  to  get  the  job  done 
(Martin,  2005)  via  empowerment  (Morton,  2002).  However,  this 
cohort  despises  micromanagement,  becomes  irritated  with  laziness. 


Analysis  of  Generation  Y  Workforce  Motivation  Using  Multiattribute  Utility  Theory 


January  2011 


and  abhors  slowness  (Weingarten,  2009).  To  some,  Generation 
Y’s  work  values  and  attributes  paint  a  picture  of  being  high  main¬ 
tenance  (Hira,  2007).  Twenge,  Zhang,  and  Im  (2004)  describe 
Generation  Y  as  having  a  “high  external  locus  of  control,”  which 
further  exemplifies  their  confidence  inside  and  outside  of  the  work¬ 
place.  However,  Crumpacker  and  Crumpacker  (2007)  note  the  need 
for  constant  approval  and  highlight  Generation  Y’s  emotionally 
needy  personality. 

A  heightened  government  retirement  of  the  Baby  Boomers  is 
almost  certain  in  the  next  several  years,  which  will  leave  employ¬ 
ment  gaps  that  Generation  X  and  Y  must  fill.  Barr  (2007,  p.  D01) 
reports  approximately  60  percent  of  the  1.8  million  government 
employees  will  be  eligible  to  retire  over  the  next  9  years.  The  Office 
of  Personnel  Management  expects  many  of  the  Baby  Boomers 
(about  40  percent)  to  retire  from  the  government.  Retention  of  the 
newly  hired  Generation  Y  workforce  is  critical  to  the  preservation 
and  existence  of  the  civilian  government  workforce. 

In  the  analysis  discussed  in  this  article,  18  government  workers, 
comprising  six  each  of  Generation  X,  Generation  Y,  and  the  Baby 
Boomers  Generation,  were  surveyed  regarding  five  motivational  fac¬ 
tors  according  to  importance  and  level  of  happiness.  The  survey  was 
designed  to  provide  insight  on  the  overall  average  job  satisfaction 
of  each  respondent  (how  happy  each  respondent  is  with  their  job 
compared  to  the  average  of  all  respondents);  the  overall  average 
job  satisfaction  of  each  generation  (how  happy  each  generation  is 
with  their  jobs  compared  to  the  average  of  all  generations);  normal¬ 
ized  average  importance  for  each  generation  (how  each  generation 
values  the  five  motivational  factors  converted  to  a  single  scale); 
average  level  of  happiness  for  each  generation  (how  each  genera¬ 
tion  is  satisfied  with  their  current  jobs  based  on  the  five  motivational 
factors);  the  overall  average  utility  (how  all  generations  combined 
express  value  and  satisfaction  for  each  of  the  five  motivational  fac¬ 
tors);  and  average  attribute  utility  for  each  generation  (how  each 
generation  expresses  value  and  satisfaction  for  each  of  the  five 
motivational  factors).  The  research  questions  that  this  study  seeks 
to  answer  follow. 

Research  Questions 

Using  the  previous  research  on  generational  life  events  coupled 
with  work  values  and  attitudes,  the  following  research  questions 
were  generated  for  analysis  in  this  study: 


No.  1.  Does  Generation  Y  assign  different  levels  of  importance  to 
the  five  motivational  factors  than  Generation  X  and  Baby  Boomers? 
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No.  2.  Does  Generation  Y  assign  different  ieveis  of  happiness  to 
the  five  nnotivationai  factors  than  Generation  X  and  Baby  Boomers; 
and  \A/hich  of  these  factors  is  ranked  the  highest  across  generations? 

No.  3.  Does  Generation  Y’s  average  attribute  utiiity  of  the  five 
motivationai  factors  differ  from  Generation  X  and  Baby  Boomers? 

Method 

Participants.  Government  \A/orkers,  six  in  each  of  the  three  age 
groups  categorized  by  Generation  Y  (born  bet\A/een  1980  and 
2000),  Generation  X  (born  bet\A/een  1965  and  1979),  and  Baby 
Boomers  (born  bet\A/een  1946  and  1964),  \A/ho  \A/ork  at  Navai  Sea 
Systems  Command,  Virginia  Beach  Detachment,  were  seiected  at 
random  by  the  detachment  security  manager.  Aii  18  respondents 
were  given  an  unseaied  enveiope  that  inciuded  a  cover  ietter  and 
an  identicai  three-page  survey.  Aii  participants  were  asked  to 
voiuntariiy  compiete  the  anonymous  survey  and  return  the  enveiope 
seaied  to  ensure  confidentiaiity.  T\A/eive  respondents  were  maie  (67 
percent),  and  six  were  femaie  (33  percent). 

The  mean  age  of  the  survey  respondents  was  36.56  (standard 
deviation  =  11.08).  Deeper  examination  into  respondent  demograph¬ 
ics  sho\A/s  13  people  (72  percent)  had  completed  either  a  bachelor’s 
or  master’s  degree. 

Materials  and  Procedure.  The  motivational  factor  survey  was 
arranged  \A/ith  six  demographic  questions,  one  motivational 
factors’  ranking  question,  and  one  level  of  happiness  question  for 
a  total  of  eight  questions.  The  demographic  set  (questions  1-6) 
consisted  of:  age,  gender,  job  classification  (either  management  or 
nonmanagement),  occupational  category  (government-designated 
categories  based  on  the  type  of  job  a  person  has),  highest  education 
completed,  and  pay  plan. 

The  motivational  factors  ranking  (question  7)  presented  the  five 
motivational  factors  and  asked  the  respondent  to  rank  them  accord¬ 
ing  to  importance.  Each  factor  was  given  a  bounded  definition 
unique  to  \A/orking  \A/ithin  a  government  context.  Factor  1  (respon¬ 
sibilities)  was  defined  as  the  value  given  to  all  responsibilities  inside 
the  office  and  \A/hile  on  government  travel.  Factor  2  (compensation) 
was  defined  as  the  value  of  the  total  government  compensation 
package,  \A/hich  includes  salary,  pension,  retirement  plan,  annual 
bonuses,  cost  of  living  increases,  etc.  Factor  3  (\A/ork  environment) 
was  defined  as  the  value  given  to  the  job  location,  people  \A/orking 
in  the  location,  and  physical  \A/ork  environment.  Factor  4  (advance¬ 
ment  potential)  was  defined  as  the  value  given  to  a  career  path 
clearly  defined  for  advancement.  Factor  5  (free  time)  was  defined 
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as  the  value  given  to  the  annount  of  free  tinne  away  from  work.  Free 
time  is  allocated  by  the  following  means:  compressed  work  sched¬ 
ule,  accrued  sick  days,  accrued  annual  days  off,  and  the  number  of 
holidays  given. 

The  level  of  happiness  (question  8)  consisted  of  each  respon¬ 
dent  ranking  the  level  of  happiness  in  their  current  position  using 
each  of  the  five  motivational  factors. 


Motivational  Factor  Rankings.  Motivational  factor  rankings  were 
determined  by  each  respondent  in  their  survey.  Each  respondent 
was  given  a  maximum  of  100  points  to  distribute  among  each  of 
the  five  motivational  factors.  The  more  points  the  participant  gave 
to  a  particular  factor,  the  more  they  valued  that  factor. 


Level  of  Happiness  Rankings.  Level  of  happiness  rankings  were 
determined  by  each  respondent  in  their  survey.  Each  respondent 
was  asked  to  rank  the  five  motivational  factors  based  on  their 
current  position.  The  format  chosen  was  a  10-point  Likert  scale  (1  = 
being  extremely  dissatisfied  and  10  =  being  extremely  satisfied). 


Procedure.  The  detachment’s  security  manager  handed  each 
respondent  an  open  envelope,  with  a  cover  letter  and  an  identical 
survey.  Participants  were  notified  in  writing  that  their  completion 
of  the  survey  indicated  their  consent  to  participate  in  this  study. 
Respondents  were  told  if  they  had  any  questions  regarding  the 
survey  to  direct  them  to  the  security  manager.  The  surveys  were 
not  traceable  to  the  survey  respondent,  and  the  deadline  to  finish 
was  1  week.  Once  completed,  the  surveys  were  to  be  placed  back 
in  the  envelope,  sealed,  and  returned  to  the  detachment  security 
manager.  The  security  manager  collected  all  18  surveys,  and  they 
were  returned  to  the  primary  author. 


Results 

Analysis  focused  on  respondents’  values  for  importance  and 
level  of  happiness  for  each  of  the  five  motivational  factors.  For 
initial  data  reduction  and  ease  of  calculation,  respondents’  ages 
were  grouped  together  by  their  generation,  as  defined  earlier  in 
this  article.  The  motivational  factors  (MF)  were  then  normalized  as 
shown  in  Equation  1  to  ensure  that  each  factor  could  be  evaluated 
on  a  100-point  scale  and  compared  with  one  another: 


MF  =  100 


X  —  OBJ  ^ 


OBJ  -OBJ 


(1) 
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OBJ  .  is  the  minimum  respondent  vaiue  (5),  OBJ  is  the  maxi- 
mum  respondent  vaiue  (50),  and  x  is  the  individuai  respondent’s 
vaiue. 

Overaii  utiiity,  denoted  as  average  job  satisfaction  (AJS),  \A/as 
then  caicuiated  using  a  traditionai  \A/eighted  sum  approach,  \A/hereby 
each  MF  is  muitipiied  by  its  reiative  importance  (ievei  of  happiness 
[LOH]),  as  sho\A/n  in  Equation  2. 

5 

AJS  =  DMF^LOH^  (2) 

k-l 


The  next  step  was  to  anaiyze  each  generation  separately  and 
average  their  respective  job  satisfaction.  Figure  1  depicts  the  overall 
average  job  satisfaction  utility  for  each  age  group.  These  averages 
were  also  used  in  computing  the  overall  job  satisfaction  utility  for 
the  entire  group  of  respondents  (average  =  1256.17,  denoted  by  the 
black  dashed  line  sho\A/n  in  Figure  1).  The  job  satisfaction  level  was 
compared  to  the  average  job  satisfaction  utility  for  the  entire  group. 

The  results  of  Figure  1  sho\A/  Generation  Y  and  Baby  Boom¬ 
ers  are  \A/ell  above  the  average  job  satisfaction  of  all  respondents. 
Conversely,  it  sho\A/s  Generation  X  is  \A/ell  belo\A/  the  average  job 
satisfaction  of  all  respondents.  Possibly,  the  Baby  Boomers  felt 
more  comfortable  \A/ith  being  honest  and  Generation  Y  shaded  their 
opinions  some\A/hat. 


FIGURE  1.  OVERALL  AVERAGE  JOB  SATISFACTION  UTILITY  FOR 
EACH  AGE  GROUP 
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Question  No.  1.  Does  Generation  Y  assign  different  ieveis  of 
importance  to  the  five  motivationai  factors  than  Generation  X  and 
Baby  Boomers? 

To  determine  whether  Generation  Y  assigns  different  ieveis  of 
importance,  the  data  were  anaiyzed  using  a  two-taiied  hypothesis 
test  at  a  0.10  significance  ievei.  Generation  Y  resuits  were  com¬ 
pared  to  Generation  X,  and  then  Baby  Boomers  for  a  total  of  10 
tests.  Of  those  10,  five  were  statistically  significant  and  therefore 
reported.  Figure  2  shows  the  normalized  average  importance  for 
each  generation. 


FIGURE  2.  NORMALIZED  AVERAGE  IMPORTANCE  FOR  EACH 
GENERATION 
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Generation  Y  views  responsibilities  as  much  less  important  than 
Generation  X  and  Baby  Boomers  and  least  important  of  all  the  moti¬ 
vational  factors.  These  results  are  statistically  significant. 

Generation  Y  ranked  compensation  as  less  important  than 
Generation  X  and  Baby  Boomers.  This  was  expected,  but  only  the 
comparison  between  Generation  Y  and  Baby  Boomers  is  statisti¬ 
cally  significant. 

Generation  Y  ranked  advancement  potential  higher  than  Gen¬ 
eration  X  and  Baby  Boomers.  Again,  the  results  between  Generation 
Y  and  Baby  Boomers  are  statistically  significant. 

Generation  Y  ranked  free  time  higher  than  Generation  X  and 
Baby  Boomers.  This  time  the  results  between  Generation  Y  and  X 
are  statistically  significant. 
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Question  No.  2.  Does  Generation  Y  assign  different  ieveis  of 
happiness  to  the  five  motivationai  factors  than  Generation  X  and 
Baby  Boonners,  and  \A/hich  of  these  factors  is  ranked  the  highest 
across  generations? 

The  data  \A/ere  anaiyzed  using  a  t\A/o-tailed  hypothesis  test  at 
a  0.10  significance  ievei.  Generation  Y  resuits  \A/ere  compared  to 
Generation  X,  and  then  Baby  Boomers  for  a  total  of  10  tests.  Of 
those  10,  two  were  statistically  significant  and  therefore  reported. 
Figure  3  sho\A/s  the  average  level  of  happiness  for  each  generation. 


FIGURE  3.  AVERAGE  LEVEL  OF  HAPPINESS  FOR  EACH 
GENERATION 
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Generation  Y  is  satisfied  \A/ith  their  current  advancement  poten¬ 
tial  in  the  government  more  than  Generation  X  and  Baby  Boomers. 
Ho\A/ever,  only  the  results  bet\A/een  Generation  Y  and  X  are  statis¬ 
tically  significant.  These  results  sho\A/  that  Generation  Y  is  very 
satisfied  \A/ith  their  current  advancement  potential  \A/ithin  the  Fed¬ 
eral  Government. 

Generation  Y  is  currently  satisfied  \A/ith  their  current  free  time 
more  than  Generation  X  and  Baby  Boomers.  Again,  the  results 
bet\A/een  Generation  Y  and  X  are  only  statistically  significant.  These 
results  show/  that  Generation  Y  is  very  satisfied  \A/ith  their  current 
free  time  \A/ithin  the  Federal  Government. 

Results  of  all  three  generations’  motivational  factors  were  aver¬ 
aged  from  the  average  attribute  utility  for  each  generation  and 
plotted  in  Figure  4.  Compensation  was  the  highest,  \A/ith  advance¬ 
ment  potential  being  the  lo\A/est  motivational  factor. 
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FIGURE  4.  AVERAGE  MOTIVATIONAL  FACTORS 
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FIGURE  5.  AVERAGE  ATTRIBUTE  UTILITY  FOR  EACH  GENERATION 
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Question  No.  3.  Does  Generation  Y’s  average  attribute  utiiity  of  the 
five  nnotivationai  factors  differ  from  Generation  X  and  Baby  Boomers? 

The  data  were  analyzed  using  a  two-tailed  hypothesis  test  at 
a  0.10  significance  level.  Generation  Y  results  were  compared  to 
Generation  X,  then  Baby  Boomers  for  a  total  of  10  tests.  Of  those 
10,  four  were  statistically  significant  and  therefore  only  reported. 
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Figure  5  shows  the  overall  average  job  satisfaction  utility  based  on 
each  attribute  for  each  generation. 

Generation  Y’s  average  attribute  utility  for  connpensation  was 
less  than  Baby  Boonners,  which  was  statistically  significant,  but 
slightly  nnore  than  Generation  X,  which  was  not  significant. 

Generation  Y’s  average  attribute  utility  for  advancennent  poten¬ 
tial  was  much  higher  than  both  Generation  X  and  Baby  Boomers. 
Both  results  were  statistically  significant. 

Generation  Y’s  average  attribute  utility  for  free  time  was  also 
higher  for  Generation  X  and  Baby  Boomers,  although  the  compari¬ 
son  to  Generation  X  was  only  statistically  significant. 

Discussion 

This  analysis  aimed  to  investigate  if  Generation  Y  assigns  differ¬ 
ing  levels  of  workplace  motivation  and  happiness  than  Generation  X 
and  Baby  Boomers  in  a  federal  government  context.  Three  research 
questions  were  developed  based  on  the  literature  review:  (1)  Does 
Generation  Y  assign  different  levels  of  importance  to  the  five  moti¬ 
vational  factors  than  Generation  X  and  Baby  Boomers?  (2)  Does 
Generation  Y  assign  different  levels  of  happiness  to  the  five  motiva¬ 
tional  factors  than  Generation  X  and  Baby  Boomers?  and  (3)  Does 
Generation  Y’s  average  attribute  utility  of  the  five  motivational  fac¬ 
tors  differ  from  Generation  X  and  Baby  Boomers? 

The  results  of  the  first  research  question  would  be  a  tentative 
yes.  Generation  Y  has  a  statistically  significant  difference  in  four  of 
the  five  motivational  factors  pertaining  to  level  of  importance.  This 
shows  Generation  Y  does  have  varying  levels  of  importance  for  four 
of  the  five  motivational  factors  when  compared  with  Generation  X 
and  Baby  Boomers. 

The  low  values  Generation  Y  attributes  to  the  responsibilities’ 
motivational  factor  are  of  intense  concern.  One  possible  explanation 
may  be  that  the  government  is  not  providing  enough  responsibilities 
to  fully  engage  Generation  Y.  Another  possible  explanation  may  be 
that  Generation  Y  is  not  happy  with  their  current  responsibilities, 
and  this  has  impacted  their  responses  to  what  motivates  them. 

Generation  Y  ranks  compensation  as  the  highest  motivational 
factor  but  not  by  much  over  the  other  factors.  The  importance 
ranks  much  less  for  Baby  Boomers,  and  this  response  is  expected. 
The  reason  is  the  Baby  Boomers  are  nearing  retirement  age  and  are 
trying  to  reach  their  maximum  earning  potential,  which  dictates  the 
amount  they  will  receive  from  their  pension.  Overall,  Generation  Y 
places  a  much  higher  importance  on  advancement  potential  and 
free  time  than  the  other  generations. 
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The  answer  to  Question  No.  2  is  a  cautious  yes.  Aithough  two 
of  the  10  possibie  combinations  are  statisticaiiy  significant,  two 
(advancement  potentiai  and  free  time)  do  provide  some  insight.  The 
two  highest  importance  ieveis  over  the  other  generations,  discussed 
eariier,  are  advancement  potentiai  and  free  time,  which  corresponds 
with  the  ievei  of  happiness  caicuiations.  Not  oniy  does  Generation 
Y  regard  advancement  potentiai  and  free  time  as  very  important, 
but  they  are  content  with  their  ieveis  of  both  motivational  factors. 

The  results  of  Question  No.  3  are  also  a  tentative  yes.  Advance¬ 
ment  potential  and  free  time  are  emerging  as  the  most  diverse 
attributes  compared  to  Generation  X  and  Baby  Boomers.  Based 
on  the  literature.  Generation  Y  proactively  plans  their  professional 
development  and  expects  to  achieve  it  within  the  federal  govern¬ 
ment.  The  majority  of  Generation  Y  research  is  done  on  the  work/ 
life  balance  factor.  Research  points  to  this  new  generation  aspiring 
to  attain  this  balance  in  their  everyday  lives.  The  results  presented 
here  promote  this  same  idea. 

Conclusions 

Questions  may  be  raised  about  the  sample  size,  concise  ques¬ 
tion  set,  and  significance  level  used.  A  much  larger  sample  size  and 
more  extensive  survey  are  needed  to  gain  an  in-depth  understand¬ 
ing  of  this  generation.  The  authors  plan  to  expand  the  participant 
pool  in  the  near  future  to  include  a  statistically  significant  number 
of  respondents.  The  expectation  is  that  the  survey  and  results 
(although  limited  due  to  small  sample  size)  described  in  this  article, 
coupled  with  the  literature  review,  will  begin  to  unveil  what  Genera¬ 
tion  Y  expects  from  a  long  and  prosperous  career  in  federal  civilian 
service.  This  can  help  management  in  aligning  corporate  incentives 
to  motivate  Generation  Y  workers,  not  only  by  compensation  but 
by  the  other  motivational  factors. 

The  federal  government’s  workforce  climate  is  shifting,  and 
conducting  internal  studies  allows  management  to  be  more  aware 
and  able  to  adapt  to  emerging  situations.  This  study  provides  the 
initial  basis  for  conducting  more  detailed  studies  specific  to  the 
federal  government.  The  government  can  be  in  the  forefront  of 
understanding  and  retaining  Generation  Y  by  conducting  research, 
validating  results  based  on  proven  mathematical  techniques,  and 
slowly  changing  the  retention  landscape  with  these  results.  By 
motivating  Generation  Y  using  the  outlined  factors,  governmental 
managers  can  tailor  retention  plans  specific  to  this  generation  to 
ensure  a  sustainable  workforce  for  the  future. 
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APPLICATION  OF 
REAL  OPTIONS  THEORY 
TO  DoD  SOFTWARE 
ACQUISITIONS 

f  Capt  Albert  Olagbemiro,  USAFR,  Johnathan  Mun, 
and  Man-Tak  Shing 

The  traditional  real  options  valuation  methodology,  when 
enhanced  and  properly  formulated  around  a  proposed  or 
existing  software  investment  employing  the  spiral  develop¬ 
ment  approach,  provides  a  framework  for  guiding  software 
acquisition  decision  making  by  highlighting  the  strategic 
importance  of  managerial  flexibility  in  managing  risk  and 
balancing  a  customer’s  requirements  within  cost  and  schedule 
constraints.  This  article  discusses  and  describes  how  an  inte¬ 
grated  risk  management  framework,  based  on  real  options 
theory,  could  be  used  as  an  effective  risk  management  tool 
to  address  the  issue  of  requirements  uncertainty  as  it  relates 
to  software  acquisition  and  guide  the  software  acquisition 
decision-making  process. 


Keywords:  Risk  Management,  Software  Acquisition, 
Strategic  Investment,  Evolutionary  Acquisition  (EA), 
Real  Options  Theory 
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Software  is  currently  the  major  expense  in  the  acquisition  of 
software-intensive  systems  (Figure  1),  with  its  role  as  a  technology 
platform  rising  from  providing  a  mere  8  percent  of  weapons  systems 
functionality  in  1960  to  over  80  percent  of  functionality  in  2000 
(Department  of  Defense  [DoD],  2000). 


FIGURE  1.  SOFTWARE  GROWTH  IN  WEAPONS  SYSTEMS 

90 


1960  1964  1970  1975  1982  1990  2000 

Weapons  Systems  and  Year  Entered  Service 

Note.  Adapted  from  Report  of  the  Defense  Science  Board  Taskforce  on  Defense  Software, 
November  2000,  by  Defense  Science  Board,  Department  of  Defense,  pp.  11-12. 


Considering  the  immense  presence  and  ever-increasing  role 
that  software  plays  in  weapons  systems,  software  is  and  should  be 
treated  as  a  capital  investment;  accordingly,  an  approach  empha¬ 
sizing  a  strategic  investment  methodology  in  its  acquisition  is 
necessary.  This  approach  would  emphasize  the  linking  of  strategic 
program  management  decisions  to  current  and  future  unknown 
software  requirements  within  the  stipulated  parameters  of  cost,  risk, 
schedule,  and  functionality.  This  strategic  program  management 
approach  is  needed  to  align  the  software  investment  under  consid¬ 
eration  within  the  context  of  the  overall  portfolio  of  existing/planned 
software  investments  to  ensure  that  synergies  in  efficiencies  are 
leveraged  in  the  delivery  of  the  intended/desired  joint  capability. 

The  key  to  the  implementation  of  a  strategic  program  man¬ 
agement  framework  is  a  disciplined  requirements  engineering 
approach  that  embodies  a  risk  management-driven  model  in  the 
acquisition  planning  process.  This  framework  would  link  and  build 
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on  two  of  the  three  key  processes  outlined  in  the  2009  Joint  Capa¬ 
bilities  Integration  and  Developnnent  System:  requirements;  the 
acquisition  process;  and  the  Planning,  Programming,  Budgeting, 
and  Execution  System. 


Method 

Risk  management  should  be  a  consideration  that  is  addressed 
much  earlier  in  the  software  engineering  process— at  the  acquisi¬ 
tion  level— during  the  investment  decision-making  activities  prior 
to  the  commitment  to  acquire  and/or  develop  a  software  system. 
The  appropriate  risk  mitigation/reduction  strategies  or  options 
should  be  crafted  much  earlier  in  the  software  investment/acquisi¬ 
tion  process,  which  leads  to  the  real  options  approach  proposed  in 
this  article. 

Real  Options  Valuation 

Real  options  valuation  originated  from  research  performed  to 
price  financial  option  contracts  in  the  field  of  financial  derivatives. 
The  underlying  premise  of  its  suitability  and  applicability  to  software 
engineering  is  based  on  the  recognition  that  strategic  flexibility 
in  software  acquisition  decisions  can  be  valued  as  a  portfolio  of 
options  or  choices  in  real  “assets”— much  akin  to  options  on  financial 
securities  that  have  real  economic  value  under  uncertainty  (Dixit  & 
Pindyck,  1995).  In  contrast  to  financial  options,  real  options  valua¬ 
tion  centers  on  real  or  nonfinancial  assets,  and  is  valuable  because  it 
enables  the  option  holder  (the  acquisition  executive)  to  take  advan¬ 
tage  of  potential  upside  benefits  while  controlling  and  hedging  risks. 

<><x>o<><><xx><><x>o<>o<>ooc><x><x><x><x>c><x><><>x><><x><x><><>o<x><><x>ooo<x><x>^^ 

When  extended  to  a  real  asset  such  as  soetware, 

REAL  OPTIONS  COULD  BE  USED  AS  A  DECISION-MAKING 
TOOL  IN  A  DYNAMIC  AND  UNCERTAIN  ENVIRONMENT. 

<>00<><XX*<>000<X><>CxX><>00<XX'<X'OC>00<X>000<XXX><>C>OC>0<>OCxX><>0<X'00<X>0<><><XX><XX*<>C><X><X>^^ 

An  option  is  a  contract  that  confers  upon  its  holder  the  right, 
without  obligation,  to  acquire  or  dispose  of  a  risky  asset  at  a  set 
price  within  a  given  period  of  time.  When  extended  to  a  real  asset 
such  as  software,  real  options  could  be  used  as  a  decision-making 
tool  in  a  dynamic  and  uncertain  environment.  A  central  and  nec¬ 
essary  tenet  of  the  real  options  approach  is  a  requirement  for 
the  presence  of  uncertainties— an  inherent  characteristic  of  most 
software  acquisition  efforts.  Real  options  are  implicit  or  explicit 
capabilities  created  for  real  assets  that  provide  the  option  holder 
with  time-deferred  and  flexible  choices  (options)  regarding  future 
risks  or  changes  of  the  software,  and  could  explicitly  address  the 
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issue  of  software  investment  choices  for  future  capabiiities.  Through 
these  capabiiities,  the  option  hoider  may  choose  to  adjust,  reduce, 
increase,  or  abandon  the  investment  in  the  future,  thereby  stabiiiz- 
ing  returns  from  these  assets.  Prior  to  its  appiication  in  any  domain, 
the  reai-options  approach  caiis  for  the  existence  of  five  pre-con¬ 
ditions.  These  pre-conditions,  as  outiined  by  Mun  (2006),  foiiow: 

1.  A  basic  financiai  modei  must  be  created  to  evaiuate  the 
costs  and  benefits  of  the  underiying  software  asset. 

2.  Uncertainties  must  exist  during  the  software  acquisition  pro¬ 
cess;  otherwise,  the  reai  options  anaiysis  becomes  useless 
as  everything  is  assumed  to  be  certain  and  known. 

3.  The  uncertainties  surrounding  the  software  acquisition 
process  must  introduce  risks,  which  directly  impact  the 
decision-making  process. 

4.  Management  must  have  the  flexibility  or  options  to  make 
mid-course  corrections  when  actively  managing  the  project. 

5.  Management  must  be  smart  enough  to  execute  the  real 
options  when  it  becomes  optimal  to  do  so. 

Since  software  acquisition  encapsulates  the  activities  related 
to  procurement  decision  making,  development,  implementation, 
and  subsequent  maintenance,  each  of  these  pre-conditions  can 
be  directly  correlated  to  the  various  activities  associated  with  a 
software  acquisition  effort.  The  uncertainties  that  surround  these 
activities  manifest  themselves  in  the  form  of  risks  and  could  range 
from  changing  or  incomplete  requirements  or  insufficient  knowl¬ 
edge  of  the  problem  domain,  to  decisions  related  to  the  future 
growth,  technology  maturation,  and  evolution  of  the  software. 

While  risks  associated  with  large-scale  software  acquisition  have 
been  effectively  managed  through  the  application  of  stochastic 
frameworks  and  project  management  techniques,  a  framework 
based  on  the  real  options  approach  is  best  suited  for  the  DoD  acqui¬ 
sition  process  because  of  its  capacity  to  overcome  the  limitations 
of  classical  financial  analysis  techniques,  such  as  the  discounted 
cash  flow  (DCF)  or  net  present  value  (NPV)  approach,  both  of 
which  treat  projects/investments  as  passively  managed,  rather  than 
actively  managed  projects/investments,  albeit  a  gross  misrepresen¬ 
tation  of  the  norm  in  software  acquisition. 

Software  Acquisition  Uncertainties 

To  tackle  the  issue  of  uncertainties  surrounding  software  acqui¬ 
sition,  a  formal  and  distinct  uncertainty  elicitation  phase  is  proposed 
as  part  of  the  software  investment  decision-making  process  (Fig¬ 
ure  2)  to  obtain  information  on  the  relevant  uncertainties  from 
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FIGURE  2.  UNCERTAINTY  ELICITATION  MODEL 
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a  strategic  point  of  view.  Stakehoiders  in  this  phase  wouid  nor- 
nnaiiy  inciude  representatives  of  the  Joint  Requirements  Oversight 
Councii,  in  concert  with  independent  requirements  subject  matter 
experts,  to  identify  and  document  uncertainties  as  they  are  reveaied 
from  an  independent  point  of  view. 

Impiementing  an  explicit  uncertainty  elicitation  phase  would 
facilitate  the  identification  of  uncertainties  very  early  on  in  the 
acquisition  process,  so  the  necessary  steps  could  be  taken  to  either 
refine  the  requirements  to  address  the  uncertainties  or  identify 
strategic  options  to  mitigate  the  risks  posed  by  the  uncertainties. 

During  the  uncertainty  elicitation  step  in  the  model,  uncertainties 
are  captured  from  two  perspectives— the  managerial  and  technical 
perspective— as  illustrated  in  Figure  3.  Managerial  uncertainties  of 
people,  time,  functionality,  budget,  and  resources  contribute  to  both 
estimation  and  schedule  uncertainties,  which  are  considered  to  be 
pragmatic  uncertainties.'  Technical  uncertainties— incomplete,  ambi¬ 
tious,  ambiguous,  changing,  or  unstable  requirements— contribute 
to  software  specification  uncertainties,  which  lead  to  software 
design  and  implementation,  software  validation,  and  software  evo¬ 
lution  uncertainties— all  of  which  can  be  categorized  as  exhibiting 
both  Heisenberg-type^  and  Gddel-like^  uncertainties. 
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FIGURE  3.  PRECONDITION  2 
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If  the  uncertainty  cannot  be  resolved,  strategic  real  options 
could  be  developed  to  address  the  risks  posed  by  the  uncertainty, 
providing  managennent  the  flexibility  to  address  the  risks  posed  by 
the  uncertainties  \A/hen  they  beconne  revealed  at  a  later  date  during 
the  acquisition  effort. 

The  Real  Options  Valuation  Framework 

To  develop  the  appropriate  options  to  hedge  against  the  risks 
due  to  the  uncertainties  surrounding  a  soft\A/are  acquisition  effort, 
\A/e  fornnulated  a  generalized  real  options  franne\A/ork  (Figure  4) 
in  line  \A/ith  the  five  preconditions  outlined  by  Mun  (2006).  This 
proposed  frame\A/ork  consists  of  the  follo\A/ing  six  phases,  each  of 
\A/hich  explicitly  addresses  and  establishes  connpliance  \A/ith  the 
preconditions. 

Needs  Assessment  Phase 
Risk  Determination  Phase 
Options  Analysis  Phase 
Options  Valuation  Phase 
Investment  Valuation  Phase 
Execution  Phase 
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We  further  validated  the  framework  and  illustrated  its  viability, 
as  an  example,  by  applying  it  to  the  Future  Combat  Systems  Net¬ 
work  (FCSN),  the  software  component  of  the  U.S.  Army  Future 
Combat  Systems  program  (Congressional  Budget  Office,  2006, 
PP.  2-21) 


Results 

Phase  I:  Needs  Assessment 

Business  Case.  The  needs  assessment  phase  culminates  with 
the  establishment  of  a  business  case  along  with  the  associated 
financial  model.  The  financial  model  is  used  to  evaluate  the  costs 
and  benefits  of  the  underlying  software  asset  being  considered 
for  acquisition  in  compliance  with  the  first  precondition  of  the  real 
options  approach.  The  traditional  discounted  cash  flow  model  with 
a  net  present  value"^  (NPV)  is  employed  to  satisfy  this  requirement, 
and  NPV  is  computed  in  terms  of  five  high-level  determinants 
(Erdogmus  &  Vandergraaf,  2004): 
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FIGURE  4.  REAL  OPTIONS  FRAMEWORK 


Identify 

Operational  Needs 


Identify  Uncertainties  Using 
Uncertainty  Elicitation  Model 


4^ 


Determine  Financial 
Value  of  Investment 


Use  Delphi  J 
Method  1 

1 

Stochastic 

Model 

Refine 

Volatility 

Compute 

Apply  DST 

Volatility 

Evidence 

■ 

4. 

1 - 

Final  Volatility  Estimate 

£ 


Factor  Option  Costs  in 
Software  Acquisition 
Cost  Estimation 

s 

Contract  Award  j 

■ 

Note.  DST  =  Dempster-Shafer  Theory.  A  mathematical  theory  of  evidence/generalization 
of  probability  theory  where  probabilities  are  assigned  to  sets  as  opposed  to  mutually 
exclusive  and  exhaustive  propositions  termed  “singletons.”  Information  from  multiple 
sources  can  be  combined  in  the  form  of  belief  assignments,  which  serves  to  aggregate 
the  information  with  respect  to  its  constituent  parts. 


89 


Application  of  Real  Options  Theory  to  DoD  Software  Acquisitions 


January  2011 


-I  (1) 

(1+r) 

/  is  the  (initial)  development  cost  of  the  FCSN 
t  is  the  (initial)  development  time  or  time  to  deploy  the  FCSN. 

C  is  the  asset  value  of  the  FCSN  over  time  t 
M  is  the  operation  cost  of  the  FCSN  over  time  t 
r  is  the  rate  at  \A/hich  all  future  cash  flo\A/s  are  to  be  discounted 
(the  discount  rate)  \A/here  the  standard  assumption  in  [1]  is 
(C-M)  is  al\A/ays  positive. 

An  NPV  of  $6.4  trillion  was  computed  for  the  FCSN  using  esti¬ 
mated  values  of  $163.7  billion,  13  years,  and  3.0  percent  for  variables 
I,  T,  and  r  respectively  based  on  key  assumptions  in  Olagbemiro 
(2008,  pp.  121-148).®  Furthermore,  a  value  of  C-M=  $10  trillion  was 
estimated  along  the  lines  of  the  assumptions  by  Olagbemiro  (2008, 
pp.  121-148). 

Uncertainty  Identification.  Uncertainty  identification  is  the  next 
crucial  step  performed  during  the  needs  assessment  phase.  In  this 
step,  the  uncertainty  elicitation  model  is  used  as  a  mechanism  to 
identify  uncertainties.  When  applied  to  the  FCSN,  it  was  determined 
that  requirements  uncertainty  fostered  by  technological  maturation 
issues  (Government  Accountability  Office  [GAO],  2008a,  pp.  89-90) 
plagued  the  FCSN  program  from  the  onset  and  introduced  several 
other  corresponding  uncertainties.  Thus,  the  follo\A/ing  uncertainties 
were  determined  to  have  been  retroactively  predictable  \A/ithin  the 
context  of  the  proposed  real-options  frame\A/ork. 

Technical  Uncertainties 

1.  Requirements  uncertainties 

2.  Integration  uncertainties 

3.  Performance  uncertainties 
Managerial  Uncertainties 

1.  Estimation  uncertainties  (size  and  cost  of  the  soft\A/are) 

2.  Scheduling  uncertainties 

Phase  II:  Risk  Determination 

The  risk  determination  phase  consists  of  two  steps:  uncertainty 
quantification  and  vo/af/7/fy  determination. 

Uncertainty  Quantification.  Uncertainty  implies  risk;  consequently, 
uncertainty  must  be  duly  quantified  as  a  risk  factor  \A/ith  the  goal 
being  to  assign  an  appropriate  numerical  value  to  the  uncertainty. 
This  is  accomplished  by  gathering  evidence  using  historical  data 
from  previous  acquisition  efforts  that  faced  similar  risks.  In  the 
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absence  of  historical  data,  the  Delphi  nnethod®  is  suggested. 
The  objective  of  the  evidence-gathering  activity  is  to  equate/ 
approxinnate  the  soft\A/are  engineering  uncertainties  of  the  current 
soft\A/are  acquisition  effort  to  a  quantifiable  property  (risk  factor) 
based  on  historical  evidence  depicted  by  previous  soft\A/are 
acquisition  efforts.  Such  evidence-gathering  activity  is  necessary 
to  gauge  the  nnagnitude/innpact  of  the  risk  on  the  underlying  asset. 
In  our  study,  \A/hile  a  suitable  proxy  for  the  FCSN  program  was  not 
readily  available  (from  a  size  perspective,  FSCN  represented  the 
largest  soft\A/are  investment/development  effort  to  date),  data 
obtained  from  the  Joint  Strike  FighteF  (JSF)  program  (JSF  soft\A/are 
component  was  one-fifth  the  size  of  the  FCSN  program)  were 
extrapolated  and  fitted  accordingly  to  mirror  the  size  of  the  FCS. 
These  data  were  then  utilized  as  a  source  of  historical  information 
for  comparative  purposes.  The  risk  of  requirements  changes  in 
the  FCSN  program  was  estimated  to  be  12  percent  (as  opposed  to 
1.44  percent  for  the  JSF  program,  \A/hich  is  one-fifth  the  size  of  the 
FCSN  program)  using  the  Capers  Jones  formula  sho\A/n  belo\A/  (Kulk 
&  Verhoef,  2008).® 


\A/here  t  is  the  time  period  in  years  during  \A/hich  the  estimates  were 
observed. 


The  Capers  Jones  approach,  \A/hich  is  a  transposition  from 
the  financial  industry,  assumes  requirements  are  compounded 
\A/ithin  a  project  and  asserts  that  the  method  of  average  percent¬ 
age  of  change  of  the  overall  requirements  volume  lacks  information 
because  it  does  not  give  any  information  on  the  time  in  v\/hich  the 
change  occurred.  Determining  time  is  an  important,  key  factor  in 
soft\A/are  engineering  since  requirements  changes  become  more 
expensive  to  implement  the  further  we  are  into  the  soft\A/are  devel¬ 
opment  process. 

Volatility  Determination.  Volatility  is  used  to  quantify  the  effect 
of  the  risk  in  the  form  of  variations  in  the  returns  associated  \A/ith 
the  soft\A/are  investment,  and  the  accuracy  of  its  estimation  is  a 
key  factor  in  real  options  valuation  because  it  drives  the  value  of 
an  option  and  is  positively  related  to  value.  While  high  volatility 
signifies  high  risk  and  implies  a  higher  discount  rate  and  lo\A/er 
value  in  traditional  NPV  valuation,  a  high  volatility  in  real  options 
analysis  is  linked  to  high-option  value.  This  link  results  from  greater 
volatility,  \A/hich  creates  a  \A/ider  range  of  possible  future  values  of 
the  opportunity  as  the  option  \A/ould  only  be  exercised  if  the  value 
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of  the  opportunity  exceeds  the  exercise  price  (Hevert,  2007). 

Figure  5  depicts  identified  uncertainties,  which  were  fed  into  a 
Monte  Cario  nnodei— Risk  Simulator®  software— taking  into  account 
interdependencies  between  both  the  technical  and  managerial 
uncertainties  associated  with  the  software  acquisition  effort.  The 
software  emulated  all  potential  combinations  and  permutations  of 
outcomes  (i.e.,  to  determine  the  effects  of  requirements  volatility  of 
12  percent  on  integration,  performance,  scheduling,  estimation,  and 
its  overall  impact  on  the  software  acquisition  effort).  The  analysis 
indicated  that  requirements  volatility  introduced  an  overall  volatility 
of  0.0866  percent  in  the  FCSN  program.  The  volatility  of  0.0866 
percent  resulted  in  a  reduction  in  the  NPV  of  the  FCSN  program 
from  $6.4  trillion  to  $6.1  trillion.  This  reduction  in  NPV  is  a  result  of 
the  potential  of  increased  costs  in  light  of  the  risks  facing  the  FCSN 


FIGURE  5.  MODELING  SOFTWARE  ENGINEERING 
UNCERTAINTIES 
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Note.  Adapted  from  Real  Options  Analysis:  Tools  and  Techniques  for  Valuing  Strategic 
Investment  and  Decisions,  by  J.  Mun,  2006.  The  Risk  Simulator  software  was  developed 
by  Mun. 
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program,  which  ultimately  reduces  the  value  of  the  investment 
effort  from  a  financial  point  of  view. 

To  improve/refine  the  accuracy  of  the  volatility  estimates,  the 
Dempster-Shafer  Theory  of  Evidence  (DST)'°  is  employed  to  provide 
increased  belief,  partial  belief,  ignorance,  or  conflict  with  the  initial 
estimates  (Arnborg  &  Hogskolan,  2006).  This  is  accomplished  by 
establishing  “belief  functions”  that  reflect  the  “degrees  of  belief” 
between  the  revised  NPV  estimate,  computed  at  $6.1  trillion  in  light 
of  the  risks  posed  by  requirements  uncertainty  and  the  FCSN  cost 
estimates  provided  by  two  independent  sources— the  Cost  Analysis 
Improvement  Group  (CAIG)  and  the  Institute  of  Defense  Analyses 
(IDA)  (Congressional  Budget  Office,  2006). 

The  independent  belief  functions  based  on  the  CAIG  and  IDA, 
which  inferred  basic  probability  assignments  associated  with  each 
of  the  FCSN  risk  factors  (requirements,  integration,  estimation  risk, 
etc.),  were  combined  using  an  orthogonal  matrix  to  determine  the 
most  probable  beliefs  for  the  set  of  risk  factors.  Where  the  com¬ 
bined  functions  reflected  “belief”  in  our  estimates,  our  estimates 
were  considered  to  be  valid  and  were  left  untouched.  In  situations 
where  the  combined  belief  functions  reflected  conflict  with  our 
estimates,  our  estimates  were  revised  accordingly  to  reflect  the 
estimates  computed  using  the  DST  approach.  Further,  we  ran  the 
Monte  Carlo  simulation  with  the  revised  risk  estimates  again,  thus 
resulting  in  a  “refined”  volatility  of  0.0947  percent.  The  derived  vola¬ 
tility,  which  reflects  an  increase  from  the  initial  volatility  estimate  of 
0.0866  percent,  results  in  a  further  reduction  of  NPV  in  the  FCSN 
program  from  $6.1  trillion  to  $5.7  trillion.  This  reduction  implies  a 
$7  billion  shortfall  ($6.4  trillion-$5.7  trillion)  between  the  original 
and  the  refined  NPV  as  a  result  of  the  volatility  of  the  software 
investment.  Details  of  the  volatility  computation  can  be  found  in 
Olagbemiro  (2008,  pp.  121-148). 

Phase  III:  Options  Analysis 

This  phase  involves  the  identification  of  options.  Once  the  volatil¬ 
ity  of  the  software  acquisition  effort  has  been  determined,  possible 
options  could  be  identified  to  manage  the  risks  associated  with  the 
software  investment  effort  (Figure  6).  In  this  study,  three  broad 
categories  of  options  are  explored  relative  to  software  acquisition. 

1.  Expand/Growth  options 

2.  Wait/Deferment  options 

3.  Contract/Switch/Abandon  options 
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FIGURE  6.  SAMPLE  OPTIONS  TO  ADDRESS  SOFTWARE 
INVESTMENTS 

Real  Option  Real  Option 

Category  Type  Description  and  Example 


Option  to  scale  up  through  cost-effective,  sequential  investments 
as  knowledge  of  the  product  increases 

A  flexibility  option  to  switch  products,  processes,  given  a  shift  in 
underlying  price  of  input  and  output  demands 

Investment  in  proprietary  assets  of  one  industry  enables  company 
to  enter  another  industry  cost  effectively  -  Link  and  Leverage 

Delay  investment  until  more  information  or  skill  is  acquired,  e.g., 
introduction  of  new  requirements 

Shrink  or  shut  down  a  project  partway  through  if  new  information 
changes  the  expected  payoffs,  e.g.,  introduction  of  new  requirements 

Switch  to  more  cost-effective  and  flexible  assets  as  new  information 
is  obtained,  e.g.,  switch  from  custom  development  to  Commercial 
Off-the-Shelf 

Limit  scope  of  (or  abandon)  software  project  when  there  is  no  further 
potential  in  the  business  opportunity  the  software  is  meant  to  address 


Note.  Adapted  from  Rea/  Options  Analysis:  Tools  and  Techniques  for  Valuing  Strategic 
Investment  and  Decisions,  by  J.  Mun,  2006. 


the  decomposition  of  the  software  into  components,  moduies,  or 
subsystems  serves  to  introduce  fiexibiiity  from  which  the  program 
manager  couid  expioit  and  benefit.  Since  the  software  design  is  a 
key  activity  aimed  at  conceiving  how  a  software  soiution  wouid 
soive  a  particuiar  probiem,  factoring  moduiar  decomposition  into 
the  design  wouid  support  the  foiiowing  two  propositions  (Damo- 
daran,  2002,  pp.  796-815): 

1.  Some  projects  that  iook  attractive  on  a  fuii-investment  basis 
may  become  even  more  attractive  if  the  project  is  parti¬ 
tioned  or  decomposed  into  components  because  we  are 
abie  to  reduce  downside  risk  at  the  iowest  possibie  ievei. 

2.  Some  projects  that  are  unattractive  on  a  fuii-investment 
basis  may  be  vaiue-creating  if  the  firm  can  invest  in  stages. 

A  successfui  moduiar  decomposition  wouid  introduce  fiexibiiity 
into  the  acquisition  process  by  recasting  the  software  effort  as  a 
series  of  options  to  start,  stop,  expand,  or  defer  the  deveiopment  of 
a  moduie  or  subsystem  when  requirements  uncertainty  is  encoun¬ 
tered.  Note  that  the  FCSN  software  effort  has  been  decomposed 
into  six  components:  Combat  Identification,  Battie  Command  and 
Mission  Execution,  Network  Management  System,  Smaii  Unmanned 
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Ground  Vehicle,  Training  Comnnon  Component,  and  System  of  Sys¬ 
tems  Common  Operating  Environment  (GAO,  2008b,  pp.  2-31).  The 
FCSN  soft\A/are  development  effort  could  be  recast  as  a  series  of 
deferment/learning  options  and  investment/gro\A/th  options.  Such 
options  may  include  start,  stop,  scale  do\A/n  (e.g.,  staff),  reallocate 
resources,  or  resume  development  v\/hen  uncertainty  is  resolved; 
or  defer  development  in  the  face  of  requirements  uncertainty.  This 
\A/hole  strategy  is  based  on  the  correct  partitioning/decomposition 
of  the  FCSN  into  the  appropriate  systems  or  subsystems. 

To  highlight  this  strategy,  we  present  a  scenario. 

Scenario:  At  least  one  out  of  the  six  software  components  is  not  facing 
requirements  uncertainty.  In  this  scenario,  we  assume  that  of  the 
six  component  systems,  one  is  not  facing  any  form  of  uncertainty, 
\A/hile  five  of  the  soft\A/are  components  are  facing  uncertainty.  We 
proceed  to  develop  different  options  to  address  this  scenario.  For 
our  study,  we  examine  two  possible  options:  compound  option  and 
deferment  option. 

Compound  Option.  In  the  event  that  at  least  one  of  the  soft\A/are 
components  is  not  facing  requirements  uncertainty,  \A/hile  all  the 
others  are  facing  requirements  uncertainty,  an  option  could  be 
developed  to  scale  down  the  resources/staff  allocated  to  the  soft- 
\A/are  components  facing  requirements  uncertainty.  The  staff  could 
then  be  switched  to  \A/ork  on  the  soft\A/are  component  that  is  not 
facing  requirements  uncertainty,  \A/hile  the  uncertainties  in  the 
other  components  are  addressed  using  our  uncertainty  elicitation 
model.  (Note:  The  assumption  \A/ith  this  approach  is  the  soft\A/are 
component  development  effort  upon  \A/hich  the  staff  engineers 
are  being  reallocated  to  v\/ork  is  not  already  behind  schedule  and 
hence  does  not  violate  Brooks  Law.y^  If  the  development  effort  upon 
\A/hich  the  staff  are  being  assigned  to  \A/ork  is  late  (behind  schedule), 
the  number  of  staff,  experience  level,  and  role  that  the  added  staff 
\A/ould  play  in  the  soft\A/are  development  effort  must  be  taken  into 
consideration.  We  therefore  framed  the  real  options  in  this  case  as: 
an  option  to  contract  and  scale  down  from  an  uncertain  system,  an 
option  to  switch  resources  to  another  system,  and  options  to  expand 
and  scale  up  staff  assigned  to  the  development  of  a  system  not  fac¬ 
ing  uncertainty  (sho\A/n  as  Strategy  A  in  Figure  7).  This  is  essentially 
a  compound  option— an  option  \A/hose  “exercise”  is  contingent  on 
the  execution  of  the  preceding  option. 
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Deferment  Option.  In  the  event  that  five  out  of  the  six  soft\A/are  com¬ 
ponents  are  facing  requirements  uncertainty,  then  an  option  could 
be  developed  to  stop  and  defer  all  development,  including  the 
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FIGURE  7.  FCS  STRATEGY  TREE  DEPICTING  STRATEGY  A  AND  B 
FOR  GIVEN  SCENARIO 
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development  of  the  soft\A/are  component  that  is  not  facing  require¬ 
ments  uncertainty  for  a  specified  period  until  uncertainty  is  resolved 
(sho\A/n  as  Strategy  B  in  Figure  7).  This  is  an  option  to  wait  and  defer. 

Phase  IV:  Options  Valuation 

Valuation  plays  a  central  part  in  any  acquisition  analysis.  Options 
are  usually  valued  based  on  the  likelihood  of  the  execution  of  the 
options.  Several  methods  are  available  for  computing  and  valuing 
real  options,  such  as  employing  the  use  of  closed-form  models, 
partial  differential  equations,  or  lattices.  For  our  study,  \A/e  utilize 
the  binomial  approach  and  apply  risk-neutral  probabilities  as  this 
method  elicits  great  appeal  due  to  its  simplicity,  ease  of  use,  and 
the  ability  to  solve  all  forms  of  customized  real-life  options. 
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We  utilize  the  Real  Options  Super  Lattice  Solver  (SLS)  3.0  soft- 
\A/are  developed  by  Real  Options  Valuation,  Inc.,  for  the  task.  The 
basic  inputs  are  presented  in  the  Table. 


TABLE.  REAL  OPTIONS  SUPER  LATTICE  SOLVER  (SLS)  3.0  INPUTS 


Symbol 

Real  Option  on  Software 
Acquisition  Project 

Description 

S 

Value  of  Underlying  Asset 
(Asset  Price) 

Current  value  of  expected 
cash  flows  (expected  benefits 
realized  from  investing  in  the 
software  effort  [NPV]) 

K 

Exercise  Price/Strike  Price 

Price  at  which  the  created 

option  would  be  realized 
(investment  cost,  or  cost  of 
investing  in  options,  which  is  an 
estimation  of  the  likely  costs  of 
accommodating  changes) 

T 

Time-to-Expiration 

The  useful  life  of  the  option 
(time  until  the  opportunity 
disappears/maturity  date  of  the 
option  contract) 

r 

Risk-Free  Interest  Rate 

Risk-free  interest  rate  relative  to 

budget  and  schedule  (interest 
rate  on  U.S.  Treasury  bonds) 

cv 

Volatility 

Uncertainty  of  the  project 

value  and  fluctuations  in  the 
value  of  the  requirements  over 
a  specified  period  of  time 
(volatility  in  requirements, 
cost  estimation,  and  schedule 
estimation  based  on  Dempster- 
Shafer  Theory  of  Evidence) 


Strategy  A.  The  Real  Options  SLS  soft\A/are  \A/as  populated 
(Figure  8)  based  on  the  follo\A/ing  underlying  values: 

1.  Development/lmplementation  cost  of  FCSN  is  $163.7  billion. 

2.  Value  of  underlying  asset  is  $6.4  trillion. 

3.  The  risk-free  rate  is  3.0  percent. 

4.  Volatility  of  our  project  is  0.0947. 

5.  Duration  of  soft\A/are  developnnent  is  13  years. 

6.  Lattice  steps  were  set  to  300. 


Application  of  Real  Options  Theory  to  DoD  Software  Acquisitions 


January  2011 


FIGURE  8.  SCREEN  CAPTURE  OF  AUTHORS’  MODEL  IN  THE  REAL 
OPTIONS  SLS  SOFTWARE 


The  model  was  executed,  and  the  lattice  of  the  underlying  asset 
(FCSN)  (Figure  9)  as  well  as  the  options  valuation  lattice  for  Strat¬ 
egy  A  (Figure  10),  was  created.  The  terminal  values  in  our  lattices 
(apex  of  lattice)  are  the  computed  values  that  occur  at  maturity, 
while  the  intermediate  values  in  the  lattices  are  the  computations 
that  occur  at  all  periods  leading  up  to  maturity.  All  these  values  are 
computed  using  backward  induction. 

The  option  analysis  that  represents  the  value  of  the  option  under 
Strategy  A  returned  a  value  of  $6.27  trillion  (Figure  10).  The  options 
valuation  lattice  of  each  phase  under  Strategy  A  was  created  and 
values  computed  using  backward  induction,  working  backward 
from  Phase  III  to  Phase  I  to  arrive  at  the  results  depicted  in  Figure  10. 

Strategy B.  In  Strategy  B,  which  calls  for  a  “defer  and  wait”  approach, 
an  assumption  is  made  that  the  duration  for  deferment  option  would 
be  3  years.  We  set  up  our  model  (Figure  11)  using  the  same  assump¬ 
tions  used  in  Strategy  A,  but  set  the  duration  of  the  deferment 
option  to  3  years. 

The  model  is  executed  and  similar  to  Strategy  A;  the  lattice  of 
the  underlying  asset  (Figure  12)  is  generated.  In  contrast,  the  option 
analysis  returned  a  value  of  $6.25  trillion  (Figure  13). 
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FIGURE  9.  LATTICE  OF  UNDERLYING  ASSET  (FCS  NETWORK) 
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FIGURE  10.  PHASE  1  (STRATEGY  A)  OPTION  VALUATION  LATTICE 
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Phase  V:  Investment  Valuation 

Given  the  option  value  of  $6.27  trillion  under  Strategy  A,  the 
intrinsic  value  of  the  connpound  option  is  determined  to  be  $6.27 
trillion  -  $5.7  trillion  =  $570  billion.  Under  Strategy  B,  the  intrinsic 
value  of  the  deferment  option  is  determined  to  be  $6.25  trillion  - 
$5.7  trillion  =  $550  billion.  This  implies  that  under  both  Strategies 
A  and  B,  the  program  manager  should  be  willing  to  pay  no  more 
than  (and  hopefully  less  than)  the  option  premium  of  $570  billion 
and  $550  billion  respectively.  This  amount,  in  addition  to  the  initial 
investment  cost  of  $163.7  billion,  should  increase  the  chances  of 
receiving  the  initially  projected  NPV  of  $6.4  trillion  for  the  FCSN  as 
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FIGURE  n.  OPTIONS  SUPER  LATTICE  SOLVER  DEFERMENT 
MODEL 


opposed  to  the  current  $5.7  trillion  in  light  of  the  risks  caused  by 
the  uncertainties  in  five  of  the  six  soft\A/are  components.  This  pre¬ 
mium  \A/ould  also  include  the  administrative  costs  associated  \A/ith 
exercising  an  option  from  an  integrated  logistics  support  point  of 
vie\A/,  i.e.,  costs  associated  \A/ith  contractual  agreements,  softw/are 
development  retooling  costs,  costs  associated  \A/ith  infrastructure 
setup  of  the  infrastructure,  etc. 

In  analyzing  both  strategies.  Strategy  A  is  more  attractive  than 
Strategy  B.  Instead  of  \A/aiting  for  another  3  years  at  an  additional 
cost  of  up  to  $550  billion  (after  \A/hich  uncertainty  \A/ould  hopefully 
have  been  resolved)  and  then  proceeding  to  spend  $163.7  billion 
at  once  to  develop  all  six  soft\A/are  components,  the  staged-phase 
approach  in  Strategy  A  calls  for  budgeting  up  to  $570  billion  for  the 
option  up  front.  The  staged-phase  approach  also  calls  for  spending 
some  $163.7  billion  for  the  System  of  Systems  Common  Operat¬ 
ing  Environment  component,  and  then  investing  more  over  time 
as  the  requirements  are  firmed  up  for  the  other  five  components. 
Therefore,  under  these  conditions.  Strategy  A,  \A/hich  employs  the 
compound  sequential  options,  is  the  optimal  approach. 
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FIGURE  12.  STRATEGY  B  UNDERLYING  ASSET  VALUE  LATTICE 
(FCS  NETWORK) 


6460.44 

6454.37 


6448.31 

6448.31 

6442.25 

6442.25 

6436.20 

6436.20 

6436.20 

6430.15 

6430.15 

6430.15 

6424.11 

6424.11 

6424.11 

6424.11 

6418.07 

6418.07 

6418.07 

6418.07 

6412.04 

6412.04 

6412.04 

6412.04 

6412.04 

^^^■6406!o2 

6406.02 

6406.02 

6406.02 

6406.02 

|6400.00^^^B 

6400.00 

6400.00 

6400.00 

6400.00 

6400.00 

^^^■  6393^ 

6393.99 

6393.99 

6393.99 

6393.99 

6387.98 

6387.98 

6387.98 

6387.98 

6387.98 

6381.98 

6381.98 

6381.98 

6381.98 

6375.98 

6375.98 

6375.98 

6375.98 

6369.99 

6369.99 

6369.99 

6364.01 

6364.01 

6364.01 

6358.03 

6358.03 

6352.05 

6352.05 

6346.08 

6340.12 

FIGURE  13.  STRATEGY  B  OPTIONS  VALUATION  LATTICE 

UNDER  DEFERMENT 
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Phase  VI:  Execution 

The  execution  phase  deals  with  the  last  precondition  of  real 
options  valuation  theory,  which  asserts  that  decision  makers  must 
be  smart  enough  to  execute  the  real  options  when  it  becomes 
optimal  to  do  so.  The  options  premium  has  two  main  components: 
intrinsic  value  and  time  value,  both  of  which  contribute  to  the  valu¬ 
ation  of  the  underlying  software  investment.  For  example,  assuming 
that  the  contract  for  the  FCSN  includes  an  option  for  Strategy  A, 
program  managers  must  then  be  willing  to  exercise  the  compound 
sequential  option  when  they  observe  that  five  of  the  six  software 
components  are  at  risk  due  to  uncertainties. 
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Discussion 

Our  proposed  approach  addresses  the  risks  associated  with 
software-reiated  capitai  investments  by  taking  a  proactive  approach 
towards  risk  management  by  emphasizing  the  pianning  for,  and 
paying  for  risk  up  front.  This  is  not  to  say  that  risk  management 
strategies  are  not  being  adopted  today,  but  rather  highiights  a 
failure  of  management  to  take  a  strategic  approach  towards  risk 
management.  The  status  quo  emphasizes  the  employment  of  what 
is  deemed  to  be  a  “tactical”  approach  in  the  form  of  the  spiral 
development  process,  which  results  in  the  elimination/reduction  of 
much  needed  functionality  from  the  scope  of  the  software  invest¬ 
ment  effort— usually  when  the  acquisition  effort  is  already  in  the 
development  phase.  Therefore,  the  proposed  methodology  in  this 
article  would  help  address  some  of  the  limitations  of  the  spiral 
development  process  by  serving  as  a  mechanism  through  which 
the  much  desired  and  needed  planning  associated  with  the  spiral 
development  process  is  provided. 

Conclusions 

Uncertainties  associated  with  software-related  capital  invest¬ 
ments  lead  to  unnecessary  and  sometimes  preventable  risks.  As 
DoD  often  sets  optimistic  requirements  for  weapons  programs 
that  require  new  and  unproven  technologies,  the  application  of  the 
real  options  valuation  methodology  would  be  beneficial  as  it  would 
enable  the  DoD  to  incorporate  the  appropriate  strategic  options 
into  acquisition  contracts.  The  options  would  serve  as  a  contract 
between  the  software  executive  and  the  contractor  (in  the  case  of 
a  government  acquisition)  to  buy  or  sell  a  specific  capability  known 
as  the  options  on  the  underlying  project.  The  real  options  valuation 
approach  is  able  to  overcome  the  limitations  of  traditional  valuation 
techniques  by  utilizing  the  best  features  of  traditional  approaches 
and  extending  their  capabilities  under  the  auspices  of  managerial 
flexibility.  Barring  the  use  of  an  explicit  uncertainty  elicitation  phase 
as  proposed  in  our  research,  and  the  development  of  options  to 
hedge  against  the  risk— and  ultimately  execute  the  options  as  they 
appear— we  believe  the  current  acquisition  process  would  continue 
to  be  plagued  by  the  risks  of  cost  and  schedule  overruns. 

The  cost-reduction  strategy  of  reducing  testing  resources  pro¬ 
posed  by  DoD  on  the  JSF  program,  while  risky  in  itself,  still  did  not 
address  the  root  causes  of  cost-related  increases  as  identified  in 
GAO  Report  No.  08-569T  (GAO,  2008c,  pp.  2-17),  further  under¬ 
scoring  the  importance  of  a  preemptive  and  strategic  approach  of 
identifying  uncertainties  early  on  in  an  acquisition  effort  and  paying 
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for  risk  up  front.  By  ennploying  our  proposed  approach,  the  DoD 
would  be  able  to  optimize  the  value  of  their  strategic  investment 
decisions  by  evaluating  several  decision  paths  under  certain  condi¬ 
tions  to  lead  to  the  optimal  investment  strategy. 
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ENDNOTES 

1.  Pragmatic  uncertainties  are  problems  in  actually  performing  the  development  activities. 

2.  Heisenberg-type  uncertainties  occur  as  the  system  is  being  developed,  grow  during 
use,  and  exhibit  themselves  in  the  form  of  changing  requirements  due  to  unsatisfactory 
behavior  post-implementation. 

3.  Godel-like  uncertainties  occur  when  the  properties  of  a  program  cannot  be  known  from 
the  representation  because  the  software  systems  and  their  specifications  are  abstract 
models  of  the  real  world. 

4.  The  NPV  valuation  approach  is  still  utilized  because  the  real  options  approach  “builds” 
on  traditional  methods  such  as  the  NPV  by  incorporating  strategic  flexibility  in  the  form 
of  “options." 

5.  NPV  of  $6.4  trillion  is  computed  based  on:  (a)  value  of  the  FCSN  program  (future  value 
less  operating  costs,  i.e.,  sum  of  [C  -  M]  was  $10  trillion;  (b)  initial  development  cost  /  was 
$163.7  billion:  (c)  r  is  3  percent;  and  (d)  time  t  to  develop  the  FCSN  is  13  years. 

6.  The  Delphi  method  is  a  subjective  estimation  methodology  based  on  the  elicitation  of  the 
opinion  of  an  expert  or  groups  of  experts  to  guide  decision  making  by  predicting  future 
events. 

7.  At  the  time  of  this  study,  the  JSF  software  acquisition  effort  represented  the  largest 
development  effort  after  the  FCSN. 

8.  The  requirements  volatility  of  12  percent  was  computed  based  on  start  and  ending  SLOC 
(Source  Lines  of  Code)  for  the  FCSN  program.  SLOC  is  used  for  demonstration  purposes 
only.  A  more  suitable  metric  such  as  function  points  is  recommended. 

9.  The  Risk  Simulator  software  was  developed  by  Johnathan  Mun. 

10.  DST  is  a  mathematical  theory  of  evidence/generalization  of  probability  theory  where 
probabilities  are  assigned  to  sets  as  opposed  to  mutually  exclusive  and  exhaustive 
propositions  termed  “singletons.”  Information  from  multiple  sources  can  be  combined  in 
the  form  of  belief  assignments,  which  serves  to  aggregate  the  information  with  respect 
to  its  constituent  parts. 

11.  Brooks  Law  states  that  adding  people  to  a  late  project  makes  it  later. 
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Review: 

During  the  Cold  War,  the  U.S.  Navy  set  about  creating  a  stealthy  nuclear 
deterrent  against  the  Soviet  Union,  based  upon  creating  a  force  of  nuclear  sub- 
nnarines  carrying  Fleet  Ballistic  Missiles  (FBMs)  kno\A/n  as  Polaris.  Fronn  1955  until 
1960,  this  capability  was  developed  and  fielded  under  a  Special  Project  Office 
(SPO)  led  by  Adnniral  Williann  F.  Raborn. 

Sapolsky  sets  out  in  this  book  to  “describe  a  government  program  \A/hich 
\A/orked,  a  public  bureaucracy  \A/hich  was  successful”  (p.1).  As  such,  it  is  a  “suc¬ 
cess  study.”  His  basic  aim  is  to  find  out  ho\A/  a  large  government  bureaucracy 
can  successfully  manage  a  technologically  challenging,  large-scale  \A/eapons 
acquisition  program. 

Sapolsky  focuses  not  on  the  technical  accomplishments  of  the  Polaris  pro¬ 
gram,  but  on  the  political/management  success.  He  does  so  by  examining  the 
four  strategies  that  the  supporters  of  the  program  used  to  protect  and  manage 
its  resources: 

•  Differentiation— “the  attempts  of  organizations  to  establish  unchal¬ 
lengeable  claims  on  valued  resources  by  distinguishing  their  own 
products  or  programs  from  those  of  competitors”  (p.  43); 

•  Co-optation— “the  attempts  of  organizations  to  absorb  ‘...new  ele¬ 
ments  into  [its]  leadership  or  policy-determining  structure. ..as  a 
means  of  averting  threats  to  its  stability  or  existence’”  (p.  47); 

•  Managerial  Innovation— “the  attempts  of  organizations  to  achieve 
autonomy  in  the  direction  of  a  complex  and  risky  program  through 
the  introduction  of  managerial  techniques  that  appear  to  indicate 
unique  managerial  competence”  (p.  58);  and 

•  Moderation— “the  attempts  of  organizations  to  build  long-term 
support  for  their  programs  by  sacrificing  short-term  gains”  (p.  54). 

Sapolsky  attempts  to  separate  the  myths  of  the  program’s  success,  which 
have  largely  been  attributed  to  managerial  innovations  such  as  PERT  (Program 
Evaluation  and  Review  Technique),  from  the  realities  such  as  the  perceived 
strategic  need  for  the  program  and  the  management  competency  of  the  SPO, 
all  of  which  created  an  environment  that  was  highly  conducive  to  eventual  suc¬ 
cess.  He  also  shows  that  in  following  a  technical  strategy  that  did  not  seek  a 
fundamental  advance  in  the  state  of  the  art,  the  Polaris  project  was  also  able  to 
deliver  the  required  performance  on  time  and  on  cost. 

This  book,  although  30  years  out  of  print  and  describing  a  now-defunct 
weapon  system,  is  essential  reading  for  managers  and  decision  makers  who 
want  to  understand  the  critical  factors  that  drive  program  success. 
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